Re: FCC Hurricane Michael after-action report

[Seth Mattinen]

On 5/15/19 7:10 PM, Brandon Martin wrote:
I dunno how the big guys get away with it.  If I hit something, you can 
darn well bet someone's going to be on my neck immediately to shut the 
job down and pull my bond if possible.



It helps when the people in the field are like 3 subcontractors removed.

Re: FCC Hurricane Michael after-action report

[Brandon Martin]

On 5/15/19 8:51 AM, Mike Hammett wrote:
The majority of people doing locates are terrible at their job. 
(Un)fortunately, people doing the conduit installations are often 
terrible at their job as well. It's about a 50/50 split if the line was 
located correctly and the installation crew was careless or the line 
wasn't located correctly in the first places. Sometimes lines can be off 
by 10 feet


We had some issues around here where the locates were literally on the 
wrong side of the street.  Crews were hitting gas lines left and right. 
 Apparently in some cases they didn't actually LOCATE at all.  They 
just sprayed and flagged based on some ancient (in many cases 50+ year 
old) "as-built" drawings.


Didn't stop the county and cities from putting a stop to it after a 
while, though.  Probably because, as you say, it was maybe 50/50 whether 
the locates were off vs. whether the boring crew just wasn't paying 
attention.  How you can't know where an active pinger on your drill head 
is to within a few inches let alone several feet is beyond me.


I dunno how the big guys get away with it.  If I hit something, you can 
darn well bet someone's going to be on my neck immediately to shut the 
job down and pull my bond if possible.

--
Brandon Martin

ARIN v4 revocations with followup felony indictments

[Martin Hannigan]

ARIN revokes fraudulently obtained IPv4 numbers:

http://www.circleid.com/posts/20190514_735k_fraudulently_obtained_ip_addresses_have_been_revoked/

USAO indicts Micfo founder in South Carolina.

https://www.justice.gov/usao-sc/pr/charleston-man-and-business-indicted-federal-court-over-9m-fraud

Relevant threads over at ARIN on PPML.

Cheers,

-M<

Re: Cisco Crosswork Network Insights - or how to destroy a useful service

[Vasileios Kotronis]

Hello,

we would be happy to collaborate to deploy and extend the ARTEMIS 
open-source software tool


for monitoring, detection and potential automated mitigation of prefix 
hijacks,


available on GitHub at https://github.com/FORTH-ICS-INSPIRE/artemis .

Current monitoring sources include RIS live, BGPStream (classic RV + RIS 
and beta BMP support) and ExaBGP APIs to local monitors.


You are most welcome to check out the code and test, provide feedback 
and/or integrate with existing custom tools you might use.


Best regards,

Vasileios

On 15/5/19 8:58 μ.μ., Dale W. Carder wrote:

Thus spake Job Snijders (j...@ntt.net) on Wed, May 15, 2019 at 12:16:06PM +0200:

I recognise the issue you describe, and I'd like to share with you that
we're going down another road. Nowadays, RIPE NCC offers a streaming API
("RIS Live") which has the data needed to analyse and correlate BGP
UPDATES seen in the wild to business rules you as operator define.

NTT folks are working on https://github.com/nlnog/bgpalerter/ - which
relies on "RIPE RIS Live", this software should become a competitive
replacement to current BGP monitoring tools. Stay tuned, the software
will be more useful in the course of the next few weeks.

Similarly, one can integrate CAIDA's BGPStream Broker Service[1] into
their own tools.  Like bgpalerter above, working with open source or
rolling your own tools is increasingly straightforward[2] due to these
community projects.

Another viable project to keep an eye on is ARTEMIS[3] for monitoring.

Dale

[1] https://bgpstream.caida.org/data
[2] https://github.com/dwcarder/bgpwatch
[3] https://www.inspire.edu.gr/artemis/


--
===
Vasileios Kotronis
Postdoctoral Researcher, member of the INSPIRE Group
INSPIRE = INternet Security, Privacy, and Intelligence REsearch
Telecommunications and Networks Lab (TNL)
Foundation for Research and Technology - Hellas (FORTH)
Leoforos Plastira 100, Heraklion 70013, Greece
Tel: +302810391241 Office: G-060
e-mail : vkotro...@ics.forth.gr
url: http://inspire.edu.gr
===

RE: FCC Hurricane Michael after-action report

[Sean Donelan]

On Mon, 13 May 2019, frnk...@iname.com wrote:

One of my takeaways from that article was that burying fiber underground
could likely have avoided many/most of these fiber cuts, though I’m not
familiar enough with the terrain to know how feasible that is.


Nature is more powerful than humans.

In Puerto Rico and US Virgin Islands, the hurricanes severely damaged 
essentially every type of infrastructure. Buried cables, aerial cables, 
microwave towers, cellular towers, satellite dishes, solar panels, primary 
and backup power stations, access roads, accesss airports, access 
seaports, broadcast radio/tv, cable TV systems, etc. etc. etc.


All the island backbone ring buried cable systems in PR experienced 
multiple cuts due to mudslides, bridge failures, and other restoration 
activites.


Immediately after the hurricanes, essentially every infrastructure 
damage assessment was 75% or worse, with most communications 
outside plant infrastructure 95% to 100% damaged.



When there is only light to moderate damage, independent repair efforts 
are faster because you avoid lots coodination meetings.  But with the 
major damage in PR and USVI, the lack of coodination caused conflicting 
repair efforts and re-work for the first several months. It wasn't patch 
and move on, it was rebuild from scratch.

Re: BGP prefix filter list

[Tom Beecher]

At a previous company , about 10-ish years ago, had the same problem due to
equipment limitations, and wasn't able to get dollars to upgrade anything.

The most effective thing for me at the time was to start dumping any prefix
with an as-path length longer than 10. For our business then, if you were
that 'far away' , there wasn't any good reason for us to keep your route.
Following default was going to be good enough.

It's still a reasonable solution I think in a lot of cases to filter out a
lot of the unnecessary prepend messes out there today.

On Wed, May 15, 2019 at 7:45 AM Baldur Norddahl 
wrote:

> Hello
>
> This morning we apparently had a problem with our routers not handling
> the full table. So I am looking into culling the least useful prefixes
> from our tables. I can hardly be the first one to take on that kind of
> project, and I am wondering if there is a ready made prefix list or
> similar?
>
> Or maybe we have a list of worst offenders? I am looking for ASN that
> announces a lot of unnecessary /24 prefixes and which happens to be far
> away from us? I would filter those to something like /20 and then just
> have a default route to catch all.
>
> Thanks,
>
> Baldur
>
>

Re: BGP prefix filter list

[Mike Hammett]

As an eyeball network myself, you'll probably want to look at those things. You 
don't need to run a CDN to know where your bits are going. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Ca By"  
To: "Mike Hammett"  
Cc: "Dan White" , nanog@nanog.org 
Sent: Wednesday, May 15, 2019 2:14:21 PM 
Subject: Re: BGP prefix filter list 







On Wed, May 15, 2019 at 11:52 AM Mike Hammett < na...@ics-il.net > wrote: 




You can't do uRPF if you're not taking full routes. 





I would never do uRPF , i am not a transit shop, so no problem there. BCP38 is 
as sexy as i get. 








You also have a more limited set of information for analytics if you don't have 
full routes. 







Yep, i don’t run a sophisticate internet CDN either. Just pumping packets from 
eyeballs to clouds and back, mostly. 









- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 



From: "Ca By" < cb.li...@gmail.com > 
To: "Dan White" < dwh...@olp.net > 
Cc: nanog@nanog.org 
Sent: Wednesday, May 15, 2019 1:50:41 PM 




Subject: Re: BGP prefix filter list 







On Wed, May 15, 2019 at 7:27 AM Dan White < dwh...@olp.net > wrote: 


On 05/15/19 13:58 +, Phil Lavin wrote: 
>> We're an eyeball network. We accept default routes from our transit 
>> providers so in theory there should be no impact on reachability. 
>> 
>> I'm pretty concerned about things that I don't know due to inefficient 
>> routing, e.g. customers hitting a public anycast DNS server in the wrong 
>> location resulting in Geolocation issues. 
> 
>Ah! Understood. The default route(s) was the bit I missed. Makes a lot of 
>sense if you can't justify buying new routers. 
> 
>Have you seen issues with Anycast routing thus far? One would assume that 
>routing would still be fairly efficient unless you're picking up transit 
>from non-local providers over extended L2 links. 

We've had no issues so far but this was a recent change. There was no 
noticeable change to outbound traffic levels. 





+1, there is no issue with this approach. 


i have been taking “provider routes” + default for a long time, works great. 


This makes sure you use each provider’s “customer cone” and SLA to the max 
while reducing your route load / churn. 


IMHO, you should only take full routes if your core business is providing full 
bgp feeds to downstrean transit customers. 




-- 
Dan White 
BTC Broadband 
Network Admin Lead 
Ph 918.366.0248 (direct) main: (918)366-8000 
Fax 918.366.6610 email: dwh...@mybtc.com 
http://www.btcbroadband.com 

Re: BGP prefix filter list

[Ca By]

On Wed, May 15, 2019 at 11:52 AM Mike Hammett  wrote:

> You can't do uRPF if you're not taking full routes.
>

I would never do uRPF , i am not a transit shop, so no problem there. BCP38
is as sexy as i get.


> You also have a more limited set of information for analytics if you don't
> have full routes.
>
>
Yep, i don’t run a sophisticate internet  CDN either. Just pumping packets
from eyeballs to clouds and back, mostly.


>
> -
> Mike Hammett
> Intelligent Computing Solutions
> http://www.ics-il.com
>
> Midwest-IX
> http://www.midwest-ix.com
>
> --
> *From: *"Ca By" 
> *To: *"Dan White" 
> *Cc: *nanog@nanog.org
> *Sent: *Wednesday, May 15, 2019 1:50:41 PM
>
> *Subject: *Re: BGP prefix filter list
>
>
>
> On Wed, May 15, 2019 at 7:27 AM Dan White  wrote:
>
>> On 05/15/19 13:58 +, Phil Lavin wrote:
>> >> We're an eyeball network. We accept default routes from our transit
>> >> providers so in theory there should be no impact on reachability.
>> >>
>> >> I'm pretty concerned about things that I don't know due to inefficient
>> >> routing, e.g. customers hitting a public anycast DNS server in the
>> wrong
>> >> location resulting in Geolocation issues.
>> >
>> >Ah! Understood. The default route(s) was the bit I missed. Makes a lot of
>> >sense if you can't justify buying new routers.
>> >
>> >Have you seen issues with Anycast routing thus far? One would assume that
>> >routing would still be fairly efficient unless you're picking up transit
>> >from non-local providers over extended L2 links.
>>
>> We've had no issues so far but this was a recent change. There was no
>> noticeable change to outbound traffic levels.
>>
>
> +1, there is no issue with this approach.
>
> i have been taking “provider routes” + default for a long time, works
> great.
>
> This makes sure you use each provider’s “customer cone” and SLA to the max
> while reducing your route load / churn.
>
> IMHO, you should only take full routes if your core business is providing
> full bgp feeds to downstrean transit customers.
>
>
>> --
>> Dan White
>> BTC Broadband
>> Network Admin Lead
>> Ph  918.366.0248 (direct)   main: (918)366-8000
>> Fax 918.366.6610email: dwh...@mybtc.com
>> http://www.btcbroadband.com
>>
>
>

Re: BGP prefix filter list

[Mike Hammett]

You can't do uRPF if you're not taking full routes. 


You also have a more limited set of information for analytics if you don't have 
full routes. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Ca By"  
To: "Dan White"  
Cc: nanog@nanog.org 
Sent: Wednesday, May 15, 2019 1:50:41 PM 
Subject: Re: BGP prefix filter list 







On Wed, May 15, 2019 at 7:27 AM Dan White < dwh...@olp.net > wrote: 


On 05/15/19 13:58 +, Phil Lavin wrote: 
>> We're an eyeball network. We accept default routes from our transit 
>> providers so in theory there should be no impact on reachability. 
>> 
>> I'm pretty concerned about things that I don't know due to inefficient 
>> routing, e.g. customers hitting a public anycast DNS server in the wrong 
>> location resulting in Geolocation issues. 
> 
>Ah! Understood. The default route(s) was the bit I missed. Makes a lot of 
>sense if you can't justify buying new routers. 
> 
>Have you seen issues with Anycast routing thus far? One would assume that 
>routing would still be fairly efficient unless you're picking up transit 
>from non-local providers over extended L2 links. 

We've had no issues so far but this was a recent change. There was no 
noticeable change to outbound traffic levels. 





+1, there is no issue with this approach. 


i have been taking “provider routes” + default for a long time, works great. 


This makes sure you use each provider’s “customer cone” and SLA to the max 
while reducing your route load / churn. 


IMHO, you should only take full routes if your core business is providing full 
bgp feeds to downstrean transit customers. 




-- 
Dan White 
BTC Broadband 
Network Admin Lead 
Ph 918.366.0248 (direct) main: (918)366-8000 
Fax 918.366.6610 email: dwh...@mybtc.com 
http://www.btcbroadband.com 

Re: BGP prefix filter list

[Mike Hammett]

I wouldn't call it shaming the vendor. There are a ton of platforms out there 
by nearly every vendor that can't accommodate modern table sizes. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Baldur Norddahl"  
To: nanog@nanog.org 
Sent: Wednesday, May 15, 2019 1:47:24 PM 
Subject: Re: BGP prefix filter list 


My purpose is not to shame the vendor, but anyway these are ZTE M6000. We are 
currently planing to implement Juniper MX204 instead, but not because of this 
incident. We just ran out of bandwidth and brand new MX204 are cheaper than 
100G capable shelves for the old platform. 


Regards, 


Baldur 




On Wed, May 15, 2019 at 8:42 PM < mike.l...@gmail.com > wrote: 





Hello Baldur, 


What routers are you running? 


-Mike 

On May 15, 2019, at 11:22, Baldur Norddahl < baldur.nordd...@gmail.com > wrote: 






Hello 


On Wed, May 15, 2019 at 3:56 PM Mike Hammett < na...@ics-il.net > wrote: 




What is the most common platform people are using with such limitations? How 
long ago was it deprecated? 








We are a small network with approx 10k customers and two core routers. The 
routers are advertised as 2 million FIB and 10 million RIB. 


This morning at about 2 AM CET our iBGP session between the two core routers 
started flapping every 5 minutes. This is how long it takes to exchange the 
full table between the routers. The eBGP sessions to our transits were stable 
and never went down. 


The iBGP session is a MPLS multiprotocol BGP session that exhanges IPv4, IPv6 
and VRF in a single session. 


We are working closely together with another ISP that have the same routers. 
His network went down as well. 


Nothing would help until I culled the majority of the IPv6 routes by installing 
a default IPv6 route together with a filter, that drops every IPv6 route 
received on our transits. After that I could not make any more experimentation. 
Need to have a maintenance window during the night. 


These routers have shared IPv4 and IPv6 memory space. My theory is that the 
combined prefix numbers is causing the problem. But it could also be some IPv6 
prefix first seen this night, that triggers a bug. Or something else. 


Regards, 


Baldur 

Re: BGP prefix filter list

[Ca By]

On Wed, May 15, 2019 at 7:27 AM Dan White  wrote:

> On 05/15/19 13:58 +, Phil Lavin wrote:
> >> We're an eyeball network. We accept default routes from our transit
> >> providers so in theory there should be no impact on reachability.
> >>
> >> I'm pretty concerned about things that I don't know due to inefficient
> >> routing, e.g. customers hitting a public anycast DNS server in the wrong
> >> location resulting in Geolocation issues.
> >
> >Ah! Understood. The default route(s) was the bit I missed. Makes a lot of
> >sense if you can't justify buying new routers.
> >
> >Have you seen issues with Anycast routing thus far? One would assume that
> >routing would still be fairly efficient unless you're picking up transit
> >from non-local providers over extended L2 links.
>
> We've had no issues so far but this was a recent change. There was no
> noticeable change to outbound traffic levels.
>

+1, there is no issue with this approach.

i have been taking “provider routes” + default for a long time, works
great.

This makes sure you use each provider’s “customer cone” and SLA to the max
while reducing your route load / churn.

IMHO, you should only take full routes if your core business is providing
full bgp feeds to downstrean transit customers.


> --
> Dan White
> BTC Broadband
> Network Admin Lead
> Ph  918.366.0248 (direct)   main: (918)366-8000
> Fax 918.366.6610email: dwh...@mybtc.com
> http://www.btcbroadband.com
>

Re: BGP prefix filter list

[Baldur Norddahl]

My purpose is not to shame the vendor, but anyway these are ZTE M6000. We
are currently planing to implement Juniper MX204 instead, but not because
of this incident. We just ran out of bandwidth and brand new MX204 are
cheaper than 100G capable shelves for the old platform.

Regards,

Baldur


On Wed, May 15, 2019 at 8:42 PM  wrote:

> Hello Baldur,
>
> What routers are you running?
>
> -Mike
>
> On May 15, 2019, at 11:22, Baldur Norddahl 
> wrote:
>
> Hello
>
> On Wed, May 15, 2019 at 3:56 PM Mike Hammett  wrote:
>
>> What is the most common platform people are using with such limitations?
>> How long ago was it deprecated?
>>
>>
>>
> We are a small network with approx 10k customers and two core routers. The
> routers are advertised as 2 million FIB and 10 million RIB.
>
> This morning at about 2 AM CET our iBGP session between the two core
> routers started flapping every 5 minutes. This is how long it takes to
> exchange the full table between the routers. The eBGP sessions to our
> transits were stable and never went down.
>
> The iBGP session is a MPLS multiprotocol BGP session that exhanges IPv4,
> IPv6 and VRF in a single session.
>
> We are working closely together with another ISP that have the same
> routers. His network went down as well.
>
> Nothing would help until I culled the majority of the IPv6 routes by
> installing a default IPv6 route together with a filter, that drops every
> IPv6 route received on our transits. After that I could not make any more
> experimentation. Need to have a maintenance window during the night.
>
> These routers have shared IPv4 and IPv6 memory space. My theory is that
> the combined prefix numbers is causing the problem. But it could also be
> some IPv6 prefix first seen this night, that triggers a bug. Or something
> else.
>
> Regards,
>
> Baldur
>
>
>

Re: BGP prefix filter list

[mike . lyon]

Hello Baldur,

What routers are you running?

-Mike

> On May 15, 2019, at 11:22, Baldur Norddahl  wrote:
> 
> Hello
> 
>> On Wed, May 15, 2019 at 3:56 PM Mike Hammett  wrote:
>> What is the most common platform people are using with such limitations? How 
>> long ago was it deprecated?
>> 
>> 
> 
> We are a small network with approx 10k customers and two core routers. The 
> routers are advertised as 2 million FIB and 10 million RIB.
> 
> This morning at about 2 AM CET our iBGP session between the two core routers 
> started flapping every 5 minutes. This is how long it takes to exchange the 
> full table between the routers. The eBGP sessions to our transits were stable 
> and never went down.
> 
> The iBGP session is a MPLS multiprotocol BGP session that exhanges IPv4, IPv6 
> and VRF in a single session.
> 
> We are working closely together with another ISP that have the same routers. 
> His network went down as well.
> 
> Nothing would help until I culled the majority of the IPv6 routes by 
> installing a default IPv6 route together with a filter, that drops every IPv6 
> route received on our transits. After that I could not make any more 
> experimentation. Need to have a maintenance window during the night. 
> 
> These routers have shared IPv4 and IPv6 memory space. My theory is that the 
> combined prefix numbers is causing the problem. But it could also be some 
> IPv6 prefix first seen this night, that triggers a bug. Or something else.
> 
> Regards,
> 
> Baldur
> 
> 

Re: BGP prefix filter list

[Baldur Norddahl]

Hello

On Wed, May 15, 2019 at 3:56 PM Mike Hammett  wrote:

> What is the most common platform people are using with such limitations?
> How long ago was it deprecated?
>
>
>
We are a small network with approx 10k customers and two core routers. The
routers are advertised as 2 million FIB and 10 million RIB.

This morning at about 2 AM CET our iBGP session between the two core
routers started flapping every 5 minutes. This is how long it takes to
exchange the full table between the routers. The eBGP sessions to our
transits were stable and never went down.

The iBGP session is a MPLS multiprotocol BGP session that exhanges IPv4,
IPv6 and VRF in a single session.

We are working closely together with another ISP that have the same
routers. His network went down as well.

Nothing would help until I culled the majority of the IPv6 routes by
installing a default IPv6 route together with a filter, that drops every
IPv6 route received on our transits. After that I could not make any more
experimentation. Need to have a maintenance window during the night.

These routers have shared IPv4 and IPv6 memory space. My theory is that the
combined prefix numbers is causing the problem. But it could also be some
IPv6 prefix first seen this night, that triggers a bug. Or something else.

Regards,

Baldur

Re: BGP prefix filter list

[Ross Tajvar]

If you're going whitebox, I would check out Netgate's new product called
TNSR. It uses VPP for the data plane, which does all its processing in user
space, thus avoiding the inefficiencies of the kernel network stack. That's
particularly important at higher speeds like 40G or 100G.

Disclaimer: I have not tried it myself but I've only heard good things.

On Wed, May 15, 2019 at 12:01 PM Brielle Bruns  wrote:

> On 5/15/2019 9:46 AM, Hansen, Christoffer wrote:
> >
> >> 'Tik, white box Linux/BSD, etc all offer good options at varying price
> >> points.
> >
> > Any pointers and/or references, when looking into speeds *above* what is
> > possible with aggregated 10G links?
> >
>
>
> That's a good question - I've not gotten past 10G yet.
>
> Cheaply, you could get ConnectX-3 40G PCIe cards and throw them in your
> favorite Dell/HP/Supermicro/other rack mount server with your Linux/BSD
> distro of choice, or VyOS for that matter.
>
> There are instructions online on converting the IB versions of the
> Mellanox cards to their Ethernet counterparts, if you want to cut some
> cost even more.
>
> --
> Brielle Bruns
> The Summit Open Source Development Group
> http://www.sosdg.org/ http://www.ahbl.org
>

Re: Cisco Crosswork Network Insights - or how to destroy a useful service

[Dale W. Carder]

Thus spake Job Snijders (j...@ntt.net) on Wed, May 15, 2019 at 12:16:06PM +0200:
> 
> I recognise the issue you describe, and I'd like to share with you that
> we're going down another road. Nowadays, RIPE NCC offers a streaming API
> ("RIS Live") which has the data needed to analyse and correlate BGP
> UPDATES seen in the wild to business rules you as operator define.
> 
> NTT folks are working on https://github.com/nlnog/bgpalerter/ - which
> relies on "RIPE RIS Live", this software should become a competitive
> replacement to current BGP monitoring tools. Stay tuned, the software
> will be more useful in the course of the next few weeks.

Similarly, one can integrate CAIDA's BGPStream Broker Service[1] into
their own tools.  Like bgpalerter above, working with open source or
rolling your own tools is increasingly straightforward[2] due to these
community projects.  

Another viable project to keep an eye on is ARTEMIS[3] for monitoring.

Dale

[1] https://bgpstream.caida.org/data
[2] https://github.com/dwcarder/bgpwatch
[3] https://www.inspire.edu.gr/artemis/

Re: BGP prefix filter list

[Radu-Adrian Feurdean]

On Wed, May 15, 2019, at 13:44, Baldur Norddahl wrote:
> Or maybe we have a list of worst offenders? I am looking for ASN that 
> announces a lot of unnecessary /24 prefixes and which happens to be far 
> away from us? I would filter those to something like /20 and then just 
> have a default route to catch all.

Hi,

You can start here : http://www.cidr-report.org/as2.0/#Gains
You will have to do some manual work in order to identify how to optimally 
filter, but you may save some space.

But the more important questions are:
 - how long will it last after one round of clean-up ?
 - can't you afford to use default route ?

You can use tools like AS-Stats (or the more expensive and much more powerful 
alternatives) if your hardware allows it, in order to get the ASes that you 
have close to no traffic towards and leave those via default.

Or, if you can afford a dedicated internet border router, there are models that 
start getting to decent pricing level on refurbished market (a thought to 
ASR9001 that should be pretty cheap these days).

Re: Cisco Crosswork Network Insights - or how to destroy a useful service

[Douglas C. Stephens via NANOG]

I would like to point out another more straightforward ignorant UI
design decision for this new service.  The login screen assumes and
requires all Cisco.com account usernames to be email addresses.  Many
are not, especially for folks like me who have had theirs for decades.


On 5/15/2019 4:50 AM, Hank Nussbacher wrote:
> I have started to use Cisco Crosswork Network Insights which is the
> replacement for BGPmon and I am shocked at how Cisco has managed to
> destroy a useful tool.  I have had a paid 50 prefix account since the
> day BGPmon became available and helped two clients implement a 500
> prefix license over the past 4 years.  None will be buying Cisco
> Crosswork Network Insights, based on my recommendation.
> 
> I really don’t know where to begin since there is so much to dislike in
> this new GUI.  I will try to give you just a small taste but I suggest
> you request a 90 day trial license and try it out for yourself.
> 
> This was not designed by someone who deals with BGP hijacks or who
> manages a network.  It was probably given to some GUI developer with a
> minimal understanding of what the users needed.   How do I know this? 
> Take for example the main configuration menu:
> https://crosswork.cisco.com/#/configuration with the first tab of
> “prefixes”.  On that page there is *no* mention of which ASN the prefix
> is associated with.  That of course was fundamental in the BGPmon menu:
> https://portal.bgpmon.net/myprefixes.php
> 
> Or take for example its “express configuration”, where you insert an ASN
> and it automatically finds all prefixes and creates a policy.  But does
> it know the name of the ASN?  Nope.  Something again that was basic in
> BGPmon via: https://portal.bgpmon.net/myasn.php is non-existent in CNI.
> 
> Or how about the alarms one gets to an email?  Want to see how that looks?
> 
> From: Crosswork Admin [mailto:ad...@crosswork.cisco.com]
> Sent: 15 May 2019 11:39
> To: Hank Nussbacher 
> Subject: CCNI Notification
> 
> Active alarm count 1 starting at 2019-05-15 08:34:42.960762315 +
> UTC. Please click on the link for each alarm below:
> https://crosswork.cisco.com/#/alarm/ba7c5084-f05d-4c12-a17f-be9e815d6647
> 
> Compare that with what we used to get:
> 
>  
> 
> 
> Possible Prefix Hijack (Code: 10)
> 
> 
> Your prefix:  99.201.0.0/16:
> Prefix Description:   Kuku net
> Update time:  2018-08-12 17:50 (UTC)
> Detected by #peers:   140
> Detected prefix:  99.201.131.0/24
> Announced by: AS46 (BGP hijacking Ltd)
> Upstream AS:  AS11 (Clueless ISP allowing customer hijacking
> Ltd)
> ASpath:   55 44 33 11 46
> Alert details:   
> https://portal.bgpmon.net/alerts.php?details_id=830521190
> Mark as false alert:  https://portal.bgpmon.net/fp.php?aid=830521190
> 
> That is just a small sampling.  Maybe two years down the road, Cisco
> will speak to customers first before destroying a useful service.
> 
> Anyone else trying this out and feels the same or feels differently?
> 
> Disappointed,
> Hank
> 
>  
> 

-- 
Douglas C. Stephens | Network Systems Analyst
Information Technology  | Phone: (515) 294-6102
Ames Laboratory, US DOE | Email: steph...@ameslab.gov

USA to Mexico IXP Equipment Recommendations

[NJ]

Advice and/or suggestions wanted. Any input greatly appreciated.

*Scenario:*
We have run fiber from the USA to Mexico and have an exchange point in
place. Our fiber connections are dark and therefore we can use any
configuration we want on as many pairs as we want. Currently we have decided
to light two 100gig connections and they are piping across the border
already. We have 200gig going to Los Angeles (Wilshire Adjacent) and
are looking to the mexico interested community of network engineers to
identify how our peering point in Tijuana can be made to best service
the networks in the region (and abroad if they build in). Our IXP is
good, but we want to update/upgrade and put in a future-compatible
robust solution instead of what we have in there now. We are an open
IXP and are wondering how you would prefer to peer and what your
equipment recommendations are and why.  We're not afraid to spend
(some) money but we do want it to not be complete overkill however
should last at least 5 to 10 years before any upgrade would be
required. Also to note, we are running our own bandwidth over the
pipes as well so we would like to keep them separated.


*Norm*
A.A.S. R.A. Lifetime Member
*Archaeology, Astronautics and SETI Research Association*
SETI Project Lifetime Contributor

Re: BGP prefix filter list

[Mike]

On 5/15/19 7:26 AM, Dovid Bender wrote:
> You have no idea how sad and true this is. 
>
> On Wed, May 15, 2019 at 10:16 AM Jon Lewis  > wrote:
>
> On Wed, 15 May 2019, Mike Hammett wrote:
>
> > What is the most common platform people are using with such
> limitations? How long ago was it deprecated?
>
> One network's deprecated router is another network's new [bargain
> priced]
> core router.  :)
>


This is very true. I picked up a nicely equipped juniper mx240 - waa
overkill for my current operation - for far, far cheaper than anything I
could have otherwise afforded new. Absolutely killer could not be
happier, and J has won a convert. But, I find this seems to be the thing
- needing capacity/feature sets/etc just to be able to stand still, but
not having the revenue stream to actually pay new for what these vendors
want to charge for their gear/licenses/etc.


Mike-

Re: BGP prefix filter list

[Karsten Elfenbein]

Hi,

did you find 
https://labs.ripe.net/Members/emileaben/768k-day-will-it-happen-did-it-happen
? It has further links at the end as well.
If you hit the 768k issue for IPv4 you might look at IPv6 as well as
there might be a 64k limit on some tcam profiles. If there is no IPv6
in use (very sad face) there might be the option to switch to a 1m
IPv4 route profile.

Using a default route might influence Reverse Path Forwarding on the
device. But you can apply outbound ACL on upstream ports as well.

The weekly routing table report has lists of worst offenders when it
comes to de aggregation or https://www.cidr-report.org/as2.0/

Karsten

Am Mi., 15. Mai 2019 um 13:45 Uhr schrieb Baldur Norddahl
:
>
> Hello
>
> This morning we apparently had a problem with our routers not handling
> the full table. So I am looking into culling the least useful prefixes
> from our tables. I can hardly be the first one to take on that kind of
> project, and I am wondering if there is a ready made prefix list or similar?
>
> Or maybe we have a list of worst offenders? I am looking for ASN that
> announces a lot of unnecessary /24 prefixes and which happens to be far
> away from us? I would filter those to something like /20 and then just
> have a default route to catch all.
>
> Thanks,
>
> Baldur
>

Re: BGP prefix filter list

[Brielle Bruns]

On 5/15/2019 9:46 AM, Hansen, Christoffer wrote:



'Tik, white box Linux/BSD, etc all offer good options at varying price
points.


Any pointers and/or references, when looking into speeds *above* what is
possible with aggregated 10G links?




That's a good question - I've not gotten past 10G yet.

Cheaply, you could get ConnectX-3 40G PCIe cards and throw them in your 
favorite Dell/HP/Supermicro/other rack mount server with your Linux/BSD 
distro of choice, or VyOS for that matter.


There are instructions online on converting the IB versions of the 
Mellanox cards to their Ethernet counterparts, if you want to cut some 
cost even more.


--
Brielle Bruns
The Summit Open Source Development Group
http://www.sosdg.org/ http://www.ahbl.org

Re: BGP prefix filter list

[Hansen, Christoffer]

On 15/05/2019 17:28, Brielle Bruns wrote:
> Lots of good non-big vendor options these days - times have changed for
> sure.

Indeed.

> 'Tik, white box Linux/BSD, etc all offer good options at varying price
> points.

Any pointers and/or references, when looking into speeds *above* what is
possible with aggregated 10G links?

Re: BGP prefix filter list

[Brielle Bruns]

On 5/15/2019 9:10 AM, Mike Hammett wrote:
Eh...  you'll find it hard to get that past me. I know hundreds of 
self-funded ISPs that don't have route table size issues.



Lots of good non-big vendor options these days - times have changed for 
sure.


I'm running an EdgeRouter Infinity with BGP feeds for v4 and v6 at home 
- very reasonably priced router with lots of ports and functionality.


Even the old EdgeRouter Lite supported multiple BGP tables - and that 
was 7 years ago at a ~ $100 price point.  But, for sub 200 can get an 
ER4 which will do most of the things the $1000+ routers will do.


'Tik, white box Linux/BSD, etc all offer good options at varying price 
points.



--
Brielle Bruns
The Summit Open Source Development Group
http://www.sosdg.org/ http://www.ahbl.org

Re: BGP prefix filter list

[Mike Hammett]

Eh... you'll find it hard to get that past me. I know hundreds of self-funded 
ISPs that don't have route table size issues. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Jon Lewis"  
To: "Mike Hammett"  
Cc: nanog@nanog.org 
Sent: Wednesday, May 15, 2019 9:14:57 AM 
Subject: Re: BGP prefix filter list 

On Wed, 15 May 2019, Mike Hammett wrote: 

> What is the most common platform people are using with such limitations? How 
> long ago was it deprecated? 

One network's deprecated router is another network's new [bargain priced] 
core router. :) 

-- 
Jon Lewis, MCP :) | I route 
| therefore you are 
_ http://www.lewis.org/~jlewis/pgp for PGP public key_ 

Re: BGP prefix filter list

[Dovid Bender]

You have no idea how sad and true this is.

On Wed, May 15, 2019 at 10:16 AM Jon Lewis  wrote:

> On Wed, 15 May 2019, Mike Hammett wrote:
>
> > What is the most common platform people are using with such limitations?
> How long ago was it deprecated?
>
> One network's deprecated router is another network's new [bargain priced]
> core router.  :)
>
> --
>   Jon Lewis, MCP :)   |  I route
>   |  therefore you are
> _ http://www.lewis.org/~jlewis/pgp for PGP public key_
>

Re: BGP prefix filter list

[Dan White]

On 05/15/19 13:58 +, Phil Lavin wrote:

We're an eyeball network. We accept default routes from our transit
providers so in theory there should be no impact on reachability.

I'm pretty concerned about things that I don't know due to inefficient
routing, e.g. customers hitting a public anycast DNS server in the wrong
location resulting in Geolocation issues.


Ah! Understood. The default route(s) was the bit I missed. Makes a lot of
sense if you can't justify buying new routers.

Have you seen issues with Anycast routing thus far? One would assume that
routing would still be fairly efficient unless you're picking up transit
from non-local providers over extended L2 links.


We've had no issues so far but this was a recent change. There was no
noticeable change to outbound traffic levels. 


--
Dan White
BTC Broadband
Network Admin Lead
Ph  918.366.0248 (direct)   main: (918)366-8000
Fax 918.366.6610email: dwh...@mybtc.com
http://www.btcbroadband.com

Re: BGP prefix filter list

[Jon Lewis]

On Wed, 15 May 2019, Mike Hammett wrote:


What is the most common platform people are using with such limitations? How 
long ago was it deprecated?


One network's deprecated router is another network's new [bargain priced] 
core router.  :)


--
 Jon Lewis, MCP :)   |  I route
 |  therefore you are
_ http://www.lewis.org/~jlewis/pgp for PGP public key_

Re: BGP prefix filter list

[Jon Lewis]

On Wed, 15 May 2019, Baldur Norddahl wrote:


Hello

This morning we apparently had a problem with our routers not handling the 
full table. So I am looking into culling the least useful prefixes from our 
tables. I can hardly be the first one to take on that kind of project, and I 
am wondering if there is a ready made prefix list or similar?


Or maybe we have a list of worst offenders? I am looking for ASN that 
announces a lot of unnecessary /24 prefixes and which happens to be far away 
from us? I would filter those to something like /20 and then just have a 
default route to catch all.


This may be too old to be terribly useful other than as a starting point, 
but we went through essentially the same thing a little more than 10 years 
ago:


http://jonsblog.lewis.org/2008/01/19#bgp

--
 Jon Lewis, MCP :)   |  I route
 |  therefore you are
_ http://www.lewis.org/~jlewis/pgp for PGP public key_

RE: BGP prefix filter list

[Phil Lavin]

> We're an eyeball network. We accept default routes from our transit providers 
> so in theory there should be no impact on reachability.

> I'm pretty concerned about things that I don't know due to inefficient 
> routing, e.g. customers hitting a public anycast DNS server in the wrong 
> location resulting in Geolocation issues.

Ah! Understood. The default route(s) was the bit I missed. Makes a lot of sense 
if you can't justify buying new routers.

Have you seen issues with Anycast routing thus far? One would assume that 
routing would still be fairly efficient unless you're picking up transit from 
non-local providers over extended L2 links.

Re: BGP prefix filter list

[Mike Hammett]

What is the most common platform people are using with such limitations? How 
long ago was it deprecated? 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Baldur Norddahl"  
To: nanog@nanog.org 
Sent: Wednesday, May 15, 2019 6:43:30 AM 
Subject: BGP prefix filter list 

Hello 

This morning we apparently had a problem with our routers not handling 
the full table. So I am looking into culling the least useful prefixes 
from our tables. I can hardly be the first one to take on that kind of 
project, and I am wondering if there is a ready made prefix list or similar? 

Or maybe we have a list of worst offenders? I am looking for ASN that 
announces a lot of unnecessary /24 prefixes and which happens to be far 
away from us? I would filter those to something like /20 and then just 
have a default route to catch all. 

Thanks, 

Baldur 

Re: BGP prefix filter list

[Brielle]

Would also cut out anyone who uses /24s for anycast, or just general traffic 
control...

Or as you put it, an insane amount of important stuff.

Sent from my iPhone

On May 15, 2019, at 7:44 AM, Phil Lavin  wrote:

>> We recently filtered out >=/24 prefixes since we're impacted by 768k day.
> 
> What kind of network are you running? Doing such prefix filtering on an 
> eyeball network strikes me as insane - you'd be cutting off customers from 
> huge swathes of the Internet (including small companies like us) that don't 
> have large IPv4 sequential allocations.

Re: BGP prefix filter list

[Dan White]

On 05/15/19 13:44 +, Phil Lavin wrote:

We recently filtered out >=/24 prefixes since we're impacted by 768k day.


What kind of network are you running? Doing such prefix filtering on an
eyeball network strikes me as insane - you'd be cutting off customers from
huge swathes of the Internet (including small companies like us) that don't
have large IPv4 sequential allocations.


We're an eyeball network. We accept default routes from our transit
providers so in theory there should be no impact on reachability.

I'm pretty concerned about things that I don't know due to inefficient
routing, e.g. customers hitting a public anycast DNS server in the wrong
location resulting in Geolocation issues.

--
Dan White
BTC Broadband
Network Admin Lead
Ph  918.366.0248 (direct)   main: (918)366-8000
Fax 918.366.6610email: dwh...@mybtc.com
http://www.btcbroadband.com

Re: BGP prefix filter list

[Antonios Chariton]

If you have multiple transit providers and still want to be able to push 
traffic to the best path (no default route), then maybe a filter that will 
accept only AS Path 2/3 or shorter per transit provider, and a default route 
for the rest. You will get significantly less prefixes, and BGP path selection 
will work “locally”. For far away prefixes though (more than 4 ASes away), you 
will not (always) pick the best path.

> On 15 May 2019, at 16:36, Dan White  wrote:
> 
> We recently filtered out >=/24 prefixes since we're impacted by 768k day.
> I'm attaching our lightly researched list of exceptions. I'm interested in
> what others' operational experience is with filtering in this way.
> 
> Filtering /24s cut our table down to around 315K.
> 
> On 05/15/19 13:43 +0200, Baldur Norddahl wrote:
>> Hello
>> 
>> This morning we apparently had a problem with our routers not handling the 
>> full table. So I am looking into culling the least useful prefixes from our 
>> tables. I can hardly be the first one to take on that kind of project, and I 
>> am wondering if there is a ready made prefix list or similar?
>> 
>> Or maybe we have a list of worst offenders? I am looking for ASN that 
>> announces a lot of unnecessary /24 prefixes and which happens to be far away 
>> from us? I would filter those to something like /20 and then just have a 
>> default route to catch all.
>> 
>> Thanks,
>> 
>> Baldur
>> 
> 
> -- 
> Dan White
> BTC Broadband
> Network Admin Lead
> Ph  918.366.0248 (direct)   main: (918)366-8000
> Fax 918.366.6610email: dwh...@mybtc.com
> http://www.btcbroadband.com
> 

RE: BGP prefix filter list

[Phil Lavin]

> We recently filtered out >=/24 prefixes since we're impacted by 768k day.

What kind of network are you running? Doing such prefix filtering on an eyeball 
network strikes me as insane - you'd be cutting off customers from huge swathes 
of the Internet (including small companies like us) that don't have large IPv4 
sequential allocations.

Re: BGP prefix filter list

[Dan White]

We recently filtered out >=/24 prefixes since we're impacted by 768k day.
I'm attaching our lightly researched list of exceptions. I'm interested in
what others' operational experience is with filtering in this way.

Filtering /24s cut our table down to around 315K.

On 05/15/19 13:43 +0200, Baldur Norddahl wrote:

Hello

This morning we apparently had a problem with our routers not handling 
the full table. So I am looking into culling the least useful prefixes 
from our tables. I can hardly be the first one to take on that kind of 
project, and I am wondering if there is a ready made prefix list or 
similar?


Or maybe we have a list of worst offenders? I am looking for ASN that 
announces a lot of unnecessary /24 prefixes and which happens to be 
far away from us? I would filter those to something like /20 and then 
just have a default route to catch all.


Thanks,

Baldur



--
Dan White
BTC Broadband
Network Admin Lead
Ph  918.366.0248 (direct)   main: (918)366-8000
Fax 918.366.6610email: dwh...@mybtc.com
http://www.btcbroadband.com
ip prefix-list root-dns seq 1 permit 198.41.0.0/24
ip prefix-list root-dns seq 2 permit 199.9.14.0/24
ip prefix-list root-dns seq 3 permit 192.33.4.0/24
ip prefix-list root-dns seq 4 permit 199.7.91.0/24
ip prefix-list root-dns seq 5 permit 192.203.230.0/24
ip prefix-list root-dns seq 6 permit 192.5.5.0/24
ip prefix-list root-dns seq 7 permit 192.112.36.0/24
ip prefix-list root-dns seq 8 permit 198.97.190.0/24
ip prefix-list root-dns seq 9 permit 192.36.148.0/24
ip prefix-list root-dns seq 10 permit 192.58.128.0/24
ip prefix-list root-dns seq 11 permit 193.0.14.0/24
ip prefix-list root-dns seq 12 permit 199.7.83.0/24
ip prefix-list root-dns seq 13 permit 202.12.27.0/24

ip prefix-list arpa-dns seq 1 permit 199.180.182.0/24
ip prefix-list arpa-dns seq 2 permit 199.253.183.0/24
ip prefix-list arpa-dns seq 3 permit 196.216.169.0/24
ip prefix-list arpa-dns seq 4 permit 203.119.86.0/24
ip prefix-list arpa-dns seq 5 permit 193.0.9.0/24

ip prefix-list gtld-dns seq 1 permit 192.5.6.0/24
ip prefix-list gtld-dns seq 2 permit 192.33.14.0/24
ip prefix-list gtld-dns seq 3 permit 192.26.92.0/24
ip prefix-list gtld-dns seq 4 permit 192.31.80.0/24
ip prefix-list gtld-dns seq 5 permit 192.12.94.0/24
ip prefix-list gtld-dns seq 6 permit 192.35.51.0/24
ip prefix-list gtld-dns seq 7 permit 192.42.93.0/24
ip prefix-list gtld-dns seq 8 permit 192.54.112.0/24
ip prefix-list gtld-dns seq 9 permit 192.43.172.0/24
ip prefix-list gtld-dns seq 10 permit 192.48.79.0/24
ip prefix-list gtld-dns seq 11 permit 192.52.178.0/24
ip prefix-list gtld-dns seq 12 permit 192.41.162.0/24
ip prefix-list gtld-dns seq 13 permit 192.55.83.0/24

ip prefix-list common-public-dns seq 1 permit 8.8.8.0/24
ip prefix-list common-public-dns seq 2 permit 8.8.4.0/24
ip prefix-list common-public-dns seq 3 permit 199.85.126.0/24
ip prefix-list common-public-dns seq 4 permit 199.85.127.0/24
ip prefix-list common-public-dns seq 5 permit 208.67.222.0/24
ip prefix-list common-public-dns seq 6 permit 208.67.220.0/24
ip prefix-list common-public-dns seq 7 permit 8.26.56.0/24
ip prefix-list common-public-dns seq 8 permit 8.20.247.0/24
ip prefix-list common-public-dns seq 9 permit 64.6.64.0/24
ip prefix-list common-public-dns seq 10 permit 64.6.65.0/24
ip prefix-list common-public-dns seq 11 permit 1.1.1.0/24
ip prefix-list common-public-dns seq 12 permit 1.0.0.0/24

! ARIN
ip prefix-list critical-infrastructure seq 1 permit 149.112.112.0/24
ip prefix-list critical-infrastructure seq 2 permit 149.112.149.0/24
ip prefix-list critical-infrastructure seq 6 permit 192.30.45.0/24
ip prefix-list critical-infrastructure seq 9 permit 192.34.238.0/24
ip prefix-list critical-infrastructure seq 13 permit 192.42.173.0/24
ip prefix-list critical-infrastructure seq 14 permit 192.42.178.0/24
ip prefix-list critical-infrastructure seq 21 permit 192.68.130.0/24
ip prefix-list critical-infrastructure seq 22 permit 192.81.185.0/24
ip prefix-list critical-infrastructure seq 23 permit 192.82.133.0/24
ip prefix-list critical-infrastructure seq 24 permit 192.82.138.0/24
ip prefix-list critical-infrastructure seq 25 permit 192.149.62.0/24
ip prefix-list critical-infrastructure seq 26 permit 192.149.63.0/24
ip prefix-list critical-infrastructure seq 27 permit 192.149.64.0/24
ip prefix-list critical-infrastructure seq 28 permit 192.149.65.0/24
ip prefix-list critical-infrastructure seq 29 permit 192.149.66.0/24
ip prefix-list critical-infrastructure seq 30 permit 192.158.252.0/24
ip prefix-list critical-infrastructure seq 31 permit 192.228.21.0/24
ip prefix-list critical-infrastructure seq 32 permit 192.228.79.0/24
ip prefix-list critical-infrastructure seq 33 permit 192.228.92.0/24
ip prefix-list critical-infrastructure seq 34 permit 199.4.137.0/24
ip prefix-list critical-infrastructure seq 35 permit 199.4.138.0/24
ip prefix-list critical-infrastructure seq 36 permit 199.4.144.0/24
ip prefix-list critical-infrastructure seq 37 permit 199.5.26.0/24

Re: Charter and Cox contacts

[daniel]

First off, I apologize for the list. I didn't realize my replies included the 
entire original email... 
Mark, we do not run a mail server. The issue is a lot of our customers have 
their 2nd homes here, 
so when they come up they are using their email provided by their 1st home ISP. 
A number of our customers use Cox, Charter, and Spectrum and can receive email 
just fine but cannot send. 
Report from a mac book pro email client showed Cox to be blocking the IP that 
we NAT most customers through.

Message: 7
Date: Tue, 14 May 2019 11:22:41 -0700 (PDT)
From: Mark Milhollan 
To: NANOG list 
Subject: Re: Charter and Cox contacts
Message-ID: 
Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed

On Mon, 13 May 2019, Stephen Satchell wrote:
>On 5/13/19 12:11 PM, dan...@pyranah.com wrote:

>>Does anyone have contacts at Charter (Spectrum) and Cox? For some reason,
>>our IP has been blocked by them and our customers are unable to send email
>>via their charter/cox accounts. Thanks
>
>Would you be talking about port 25/tcp outbound?  Lots of ISPs will
>block port 25 as a rule;

If this is the case, that your customers cannot use your mail servers to 
relay mail then provide port 587/TCP (SUBMISSION) or even 465/TCP 
(SMTPS, deprecated).

If you mean that your mail servers can't send mail to Charter and Cox 
addresses, then indeed you need to find out what happened, fix it then 
request removal from their (and other) blacklists for which contacts 
there would be helpful (I am not one).


/mark

Daniel Temple
VP of Operations
Pyranah Communications, LLC
828-743-2470 ext.205
Pyranah.com
Like us on Facebook!

Re: Cisco Crosswork Network Insights - or how to destroy a useful service

[Patrick McEvilly]

 

 

 

https://honestnetworker.net/2019/01/31/recent-bgpmon-net-announcement/

 

 

From: NANOG  on behalf of 
Mike Hammett 
Date: Wednesday, May 15, 2019 at 8:35 AM
To: Hank Nussbacher 
Cc: "nanog@nanog.org" 
Subject: Re: Cisco Crosswork Network Insights - or how to destroy a useful 
service
Resent-From: Patrick McEvilly 

 

Cisco ruins everything they touch.



-
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com

 

From: "Hank Nussbacher" 
To: nanog@nanog.org
Sent: Wednesday, May 15, 2019 4:50:10 AM
Subject: Cisco Crosswork Network Insights - or how to destroy a useful service

I have started to use Cisco Crosswork Network Insights which is the replacement 
for BGPmon and I am shocked at how Cisco has managed to destroy a useful tool.  
I have had a paid 50 prefix account since the day BGPmon became available and 
helped two clients implement a 500 prefix license over the past 4 years.  None 
will be buying Cisco Crosswork Network Insights, based on my recommendation.

I really don’t know where to begin since there is so much to dislike in this 
new GUI.  I will try to give you just a small taste but I suggest you request a 
90 day trial license and try it out for yourself.

This was not designed by someone who deals with BGP hijacks or who manages a 
network.  It was probably given to some GUI developer with a minimal 
understanding of what the users needed.   How do I know this?  Take for example 
the main configuration menu: https://crosswork.cisco.com/#/configuration with 
the first tab of “prefixes”.  On that page there is no mention of which ASN the 
prefix is associated with.  That of course was fundamental in the BGPmon menu: 
https://portal.bgpmon.net/myprefixes.php

Or take for example its “express configuration”, where you insert an ASN and it 
automatically finds all prefixes and creates a policy.  But does it know the 
name of the ASN?  Nope.  Something again that was basic in BGPmon via: 
https://portal.bgpmon.net/myasn.php is non-existent in CNI.

Or how about the alarms one gets to an email?  Want to see how that looks?

From: Crosswork Admin [mailto:ad...@crosswork.cisco.com] 
Sent: 15 May 2019 11:39
To: Hank Nussbacher 
Subject: CCNI Notification

Active alarm count 1 starting at 2019-05-15 08:34:42.960762315 + UTC. 
Please click on the link for each alarm below: 

https://crosswork.cisco.com/#/alarm/ba7c5084-f05d-4c12-a17f-be9e815d6647

Compare that with what we used to get:

 


Possible Prefix Hijack (Code: 10)


Your prefix:  99.201.0.0/16:
Prefix Description:   Kuku net
Update time:  2018-08-12 17:50 (UTC)
Detected by #peers:   140
Detected prefix:  99.201.131.0/24
Announced by: AS46 (BGP hijacking Ltd)
Upstream AS:  AS11 (Clueless ISP allowing customer hijacking Ltd)
ASpath:   55 44 33 11 46
Alert details:    
https://portal.bgpmon.net/alerts.php?details_id=830521190
Mark as false alert:  https://portal.bgpmon.net/fp.php?aid=830521190

That is just a small sampling.  Maybe two years down the road, Cisco will speak 
to customers first before destroying a useful service.

Anyone else trying this out and feels the same or feels differently?

Disappointed,
Hank

 

 

Re: FCC Hurricane Michael after-action report

[Mike Hammett]

The majority of people doing locates are terrible at their job. 
(Un)fortunately, people doing the conduit installations are often terrible at 
their job as well. It's about a 50/50 split if the line was located correctly 
and the installation crew was careless or the line wasn't located correctly in 
the first places. Sometimes lines can be off by 10 feet. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Rich Kulawiec"  
To: nanog@nanog.org 
Sent: Tuesday, May 14, 2019 8:51:13 AM 
Subject: Re: FCC Hurricane Michael after-action report 

On Mon, May 13, 2019 at 11:48:02PM -0500, frnk...@iname.com wrote: 
> One of my takeaways from that article was that burying fiber underground 
> could likely have avoided many/most of these fiber cuts, though I???m 
> not familiar enough with the terrain to know how feasible that is. 

I suspect that may not be possible in (parts of) Florida. 

However, even in places where it's possible, fiber installation is 
sometimes miserably executed. Like my neighborhood. A couple of 
years ago, Verizon decided to finally bring FIOS in. They put in the 
appropriate calls to utility services, who dutifully marked all the 
existing power/cable/gas/etc. lines and then their contractors (or 
sub-sub-contractors) showed up. 

The principle outcome of their efforts quickly became clear, as one 
Comcast cable line after another was severed. Not a handful, not even 
dozens: well over a hundred. They managed to cut mine in three places, 
which was truly impressive. (Thanks for the extended outage, Verizon.) 
After this had gone on for a month, Comcast caught on and took the 
expedient route of just rolling a truck every morning. They'd park at 
the end of the road and just wait for the service calls that they knew 
were coming. Of course Comcast's lines were not the only victims of 
this incompetence and negligence. Amusingly, sometimes Verizon had to 
send its own repair crews for their copper lines. 

There's a lot more but let me skip to the end result. After inflicting 
months of outages on everyone, after tearing up lots of lawns, after all 
of this, many of the fiber conduits that are allegedly underground: aren't. 

---rsk 

Re: Cisco Crosswork Network Insights - or how to destroy a useful service

[Mike Hammett]

Cisco ruins everything they touch. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Hank Nussbacher"  
To: nanog@nanog.org 
Sent: Wednesday, May 15, 2019 4:50:10 AM 
Subject: Cisco Crosswork Network Insights - or how to destroy a useful service 


I have started to use Cisco Crosswork Network Insights which is the replacement 
for BGPmon and I am shocked at how Cisco has managed to destroy a useful tool. 
I have had a paid 50 prefix account since the day BGPmon became available and 
helped two clients implement a 500 prefix license over the past 4 years. None 
will be buying Cisco Crosswork Network Insights, based on my recommendation. 
I really don’t know where to begin since there is so much to dislike in this 
new GUI. I will try to give you just a small taste but I suggest you request a 
90 day trial license and try it out for yourself. 
This was not designed by someone who deals with BGP hijacks or who manages a 
network. It was probably given to some GUI developer with a minimal 
understanding of what the users needed. How do I know this? Take for example 
the main configuration menu: https://crosswork.cisco.com/#/configuration with 
the first tab of “prefixes”. On that page there is no mention of which ASN the 
prefix is associated with. That of course was fundamental in the BGPmon menu: 
https://portal.bgpmon.net/myprefixes.php 
Or take for example its “express configuration”, where you insert an ASN and it 
automatically finds all prefixes and creates a policy. But does it know the 
name of the ASN? Nope. Something again that was basic in BGPmon via: 
https://portal.bgpmon.net/myasn.php is non-existent in CNI. 
Or how about the alarms one gets to an email? Want to see how that looks? From: 
Crosswork Admin [ mailto:ad...@crosswork.cisco.com ] 
Sent: 15 May 2019 11:39 
To: Hank Nussbacher  
Subject: CCNI Notification 

Active alarm count 1 starting at 2019-05-15 08:34:42.960762315 + UTC. 
Please click on the link for each alarm below: 

https://crosswork.cisco.com/#/alarm/ba7c5084-f05d-4c12-a17f-be9e815d6647 

Compare that with what we used to get: 
 
Possible Prefix Hijack (Code: 10) 
 

Your prefix: 99.201.0.0/16: 
Prefix Description: Kuku net 
Update time: 2018-08-12 17:50 (UTC) 
Detected by #peers: 140 
Detected prefix: 99.201.131.0/24 
Announced by: AS46 (BGP hijacking Ltd) 
Upstream AS: AS11 (Clueless ISP allowing customer hijacking Ltd) 
ASpath: 55 44 33 11 46 
Alert details: https://portal.bgpmon.net/alerts.php?details_id=830521190 
Mark as false alert: https://portal.bgpmon.net/fp.php?aid=830521190 

That is just a small sampling. Maybe two years down the road, Cisco will speak 
to customers first before destroying a useful service. 
Anyone else trying this out and feels the same or feels differently? 
Disappointed, 
Hank 

Re: BGP prefix filter list

[Anderson, Charles R]

What about these ones?

https://teamarin.net/2019/05/13/taking-a-hard-line-on-fraud/

On Wed, May 15, 2019 at 01:43:30PM +0200, Baldur Norddahl wrote:
> Hello
> 
> This morning we apparently had a problem with our routers not handling 
> the full table. So I am looking into culling the least useful prefixes 
> from our tables. I can hardly be the first one to take on that kind of 
> project, and I am wondering if there is a ready made prefix list or similar?
> 
> Or maybe we have a list of worst offenders? I am looking for ASN that 
> announces a lot of unnecessary /24 prefixes and which happens to be far 
> away from us? I would filter those to something like /20 and then just 
> have a default route to catch all.
> 
> Thanks,
> 
> Baldur

Re: Cisco Crosswork Network Insights - or how to destroy a useful service

[hank]

https://bgpmon.net/wp-content/uploads/2019/01/BGPMon.net-EOL-EOS-faq.pdfOn May 15, 2019 14:52, "Mann, Jason"  wrote:
​Is BGPmon going away?



From: NANOG  on behalf of Hank Nussbacher 
Sent: Wednesday, May 15, 2019 3:50 AM
To: nanog@nanog.org
Subject: Cisco Crosswork Network Insights - or how to destroy a useful service
 


I have started to use Cisco Crosswork Network Insights which is the replacement for BGPmon and I am shocked at how Cisco has managed to destroy a useful tool. 
I have had a paid 50 prefix account since the day BGPmon became available and helped two clients implement a 500 prefix license over the past 4 years. 
None will be buying Cisco Crosswork Network Insights, based on my recommendation.
I really don’t know where to begin since there is so much to dislike in this new GUI. 
I will try to give you just a small taste but I suggest you request a 90 day trial license and try it out for yourself.
This was not designed by someone who deals with BGP hijacks or who manages a network. 
It was probably given to some GUI developer with a minimal understanding of what the users needed.  
How do I know this?  Take for example the main configuration menu:

https://crosswork.cisco.com/#/configuration with the first tab of “prefixes”. 
On that page there is no mention of which ASN the prefix is associated with. 
That of course was fundamental in the BGPmon menu: 
https://portal.bgpmon.net/myprefixes.php
Or take for example its “express configuration”, where you insert an ASN and it automatically finds all prefixes and creates a policy. 
But does it know the name of the ASN?  Nope. 
Something again that was basic in BGPmon via: 
https://portal.bgpmon.net/myasn.php is non-existent in CNI.
Or how about the alarms one gets to an email? 
Want to see how that looks?
From: Crosswork Admin [mailto:ad...@crosswork.cisco.com]

Sent: 15 May 2019 11:39
To: Hank Nussbacher 

Subject: CCNI Notification

Active alarm count 1 starting at 2019-05-15 08:34:42.960762315 + UTC. Please click on the link for each alarm below:


https://crosswork.cisco.com/#/alarm/ba7c5084-f05d-4c12-a17f-be9e815d6647

Compare that with what we used to get:
 

Possible Prefix Hijack (Code: 10)


Your prefix:  99.201.0.0/16:
Prefix Description:   Kuku net
Update time:  2018-08-12 17:50 (UTC)
Detected by #peers:   140
Detected prefix:  99.201.131.0/24
Announced by: AS46 (BGP hijacking Ltd)
Upstream AS:  AS11 (Clueless ISP allowing customer hijacking Ltd)
ASpath:   55 44 33 11 46
Alert details:    https://portal.bgpmon.net/alerts.php?details_id=830521190
Mark as false alert:  https://portal.bgpmon.net/fp.php?aid=830521190
That is just a small sampling.  Maybe two years down the road, Cisco will speak to customers first before destroying a useful service.
Anyone else trying this out and feels the same or feels differently?

Disappointed,
Hank
 

Re: Cisco Crosswork Network Insights - or how to destroy a useful service

[Job Snijders]

On Wed, May 15, 2019 at 11:52:16AM +, Mann, Jason via NANOG wrote:
> ?Is BGPmon going away?

Yes, see
https://bgpmon.net/wp-content/uploads/2019/01/BGPMon.net-EOL-EOS-faq.pdf

Kind regards,

Job

Re: Cisco Crosswork Network Insights - or how to destroy a useful service

[Mann, Jason via NANOG]

?Is BGPmon going away?


From: NANOG  on behalf of Hank Nussbacher 

Sent: Wednesday, May 15, 2019 3:50 AM
To: nanog@nanog.org
Subject: Cisco Crosswork Network Insights - or how to destroy a useful service

I have started to use Cisco Crosswork Network Insights which is the replacement 
for BGPmon and I am shocked at how Cisco has managed to destroy a useful tool.  
I have had a paid 50 prefix account since the day BGPmon became available and 
helped two clients implement a 500 prefix license over the past 4 years.  None 
will be buying Cisco Crosswork Network Insights, based on my recommendation.
I really don't know where to begin since there is so much to dislike in this 
new GUI.  I will try to give you just a small taste but I suggest you request a 
90 day trial license and try it out for yourself.
This was not designed by someone who deals with BGP hijacks or who manages a 
network.  It was probably given to some GUI developer with a minimal 
understanding of what the users needed.   How do I know this?  Take for example 
the main configuration menu: 
https://crosswork.cisco.com/#/configuration
 with the first tab of "prefixes".  On that page there is no mention of which 
ASN the prefix is associated with.  That of course was fundamental in the 
BGPmon menu: 
https://portal.bgpmon.net/myprefixes.php
Or take for example its "express configuration", where you insert an ASN and it 
automatically finds all prefixes and creates a policy.  But does it know the 
name of the ASN?  Nope.  Something again that was basic in BGPmon via: 
https://portal.bgpmon.net/myasn.php
 is non-existent in CNI.
Or how about the alarms one gets to an email?  Want to see how that looks?
From: Crosswork Admin [mailto:ad...@crosswork.cisco.com]
Sent: 15 May 2019 11:39
To: Hank Nussbacher 
Subject: CCNI Notification

Active alarm count 1 starting at 2019-05-15 08:34:42.960762315 + UTC. 
Please click on the link for each alarm below:
https://crosswork.cisco.com/#/alarm/ba7c5084-f05d-4c12-a17f-be9e815d6647
Compare that with what we used to get:


Possible Prefix Hijack (Code: 10)


Your prefix:  99.201.0.0/16:
Prefix Description:   Kuku net
Update time:  2018-08-12 17:50 (UTC)
Detected by #peers:   140
Detected prefix:  99.201.131.0/24
Announced by: AS46 (BGP hijacking Ltd)
Upstream AS:  AS11 (Clueless ISP allowing customer hijacking Ltd)
ASpath:   55 44 33 11 46
Alert details:
https://portal.bgpmon.net/alerts.php?details_id=830521190
Mark as false alert:  
https://portal.bgpmon.net/fp.php?aid=830521190
That is just a small sampling.  Maybe two years down the road, Cisco will speak 
to customers first before destroying a useful service.
Anyone else trying this out and feels the same or feels differently?
Disappointed,
Hank

BGP prefix filter list

[Baldur Norddahl]

Hello

This morning we apparently had a problem with our routers not handling 
the full table. So I am looking into culling the least useful prefixes 
from our tables. I can hardly be the first one to take on that kind of 
project, and I am wondering if there is a ready made prefix list or similar?


Or maybe we have a list of worst offenders? I am looking for ASN that 
announces a lot of unnecessary /24 prefixes and which happens to be far 
away from us? I would filter those to something like /20 and then just 
have a default route to catch all.


Thanks,

Baldur

Re: Cisco Crosswork Network Insights - or how to destroy a useful service

[Job Snijders]

On Wed, May 15, 2019 at 11:37:57AM +0100, Carlos Friaças wrote:
> It relies *exclusively* on "RIPE RIS Live", or does it also use other
> sources?

The first useful version will rely exclusively on the "RIS Live"
interface. In a later stage we can consider adding something like the
NLNOG Looking Glass data source.

Kind regards,

Job

Re: Cisco Crosswork Network Insights - or how to destroy a useful service

[Carlos Friaças via NANOG]

Hi Job, All,

It relies *exclusively* on "RIPE RIS Live", or does it also use other 
sources?


Regards,
Carlos


On Wed, 15 May 2019, Job Snijders wrote:


Hi,

I recognise the issue you describe, and I'd like to share with you that
we're going down another road. Nowadays, RIPE NCC offers a streaming API
("RIS Live") which has the data needed to analyse and correlate BGP
UPDATES seen in the wild to business rules you as operator define.

NTT folks are working on https://github.com/nlnog/bgpalerter/ - which
relies on "RIPE RIS Live", this software should become a competitive
replacement to current BGP monitoring tools. Stay tuned, the software
will be more useful in the course of the next few weeks.

Kind regards,

Job

Re: Cisco Crosswork Network Insights - or how to destroy a useful service

[Job Snijders]

Hi,

I recognise the issue you describe, and I'd like to share with you that
we're going down another road. Nowadays, RIPE NCC offers a streaming API
("RIS Live") which has the data needed to analyse and correlate BGP
UPDATES seen in the wild to business rules you as operator define.

NTT folks are working on https://github.com/nlnog/bgpalerter/ - which
relies on "RIPE RIS Live", this software should become a competitive
replacement to current BGP monitoring tools. Stay tuned, the software
will be more useful in the course of the next few weeks.

Kind regards,

Job

Cisco Crosswork Network Insights - or how to destroy a useful service

[Hank Nussbacher]

I have started to use Cisco Crosswork Network Insights which is the 
replacement for BGPmon and I am shocked at how Cisco has managed to 
destroy a useful tool.I have had a paid 50 prefix account since the day 
BGPmon became available and helped two clients implement a 500 prefix 
license over the past 4 years.None will be buying Cisco Crosswork 
Network Insights, based on my recommendation.


I really don’t know where to begin since there is so much to dislike in 
this new GUI.I will try to give you just a small taste but I suggest you 
request a 90 day trial license and try it out for yourself.


This was not designed by someone who deals with BGP hijacks or who 
manages a network.It was probably given to some GUI developer with a 
minimal understanding of what the users needed.How do I know this?Take 
for example the main configuration menu: 
https://crosswork.cisco.com/#/configuration with the first tab of 
“prefixes”.On that page there is *no* mention of which ASN the prefix is 
associated with.That of course was fundamental in the BGPmon menu: 
https://portal.bgpmon.net/myprefixes.php


Or take for example its “express configuration”, where you insert an ASN 
and it automatically finds all prefixes and creates a policy.But does it 
know the name of the ASN?Nope.Something again that was basic in BGPmon 
via: https://portal.bgpmon.net/myasn.php is non-existent in CNI.


Or how about the alarms one gets to an email?Want to see how that looks?

From: Crosswork Admin [mailto:ad...@crosswork.cisco.com]
Sent: 15 May 2019 11:39
To: Hank Nussbacher 
Subject: CCNI Notification

Active alarm count 1 starting at 2019-05-15 08:34:42.960762315 + 
UTC. Please click on the link for each alarm below:

https://crosswork.cisco.com/#/alarm/ba7c5084-f05d-4c12-a17f-be9e815d6647

Compare that with what we used to get:


Possible Prefix Hijack (Code: 10)


Your prefix:99.201.0.0/16:
Prefix Description:Kuku net
Update time:2018-08-12 17:50 (UTC)
Detected by #peers:140
Detected prefix:99.201.131.0/24
Announced by:AS46 (BGP hijacking Ltd)
Upstream AS:AS11 (Clueless ISP allowing customer hijacking Ltd)
ASpath:55 44 33 11 46
Alert 
details:https://portal.bgpmon.net/alerts.php?details_id=830521190

Mark as false alert:https://portal.bgpmon.net/fp.php?aid=830521190

That is just a small sampling.Maybe two years down the road, Cisco will 
speak to customers first before destroying a useful service.


Anyone else trying this out and feels the same or feels differently?

Disappointed,
Hank