Re: Was wrong Re: Did IPv6 between HE and Google ever get resolved?

[Jon Lewis]

I think what you were remembering is Cogent/Google and Cogent/HE are both 
IPv6 issues where the parties can't agree on peering vs transit for the v6 
relationship.


On Thu, 28 Mar 2019, David Hubbard wrote:



Oops, I was corrected that HE doesn’t have IPv6 issues with Google, not sure 
why I had that in my head.  Cogent certainly does but something had me thinking 
there’s another big name
that has the same problem.

 

David

 

From: NANOG  on behalf of David Hubbard 

Date: Thursday, March 28, 2019 at 12:40 PM
To: NANOG List 
Subject: Did IPv6 between HE and Google ever get resolved?

 

Hey all, I’ve been having bad luck searching around, but did IPv6 transit 
between HE and google ever get resolved?  Ironically, I can now get to them 
cheaply from a location we
currently have equipment that has been Cogent-only, so if it fixes the IPv6 
issue I’d like to make the move.  Anyone peer with HE in general and want to 
share their experience
offlist?  With the price, if they’re a good option, I’d consider rolling them 
in to other locations where we have redundancy already, so the v6 isn’t as big 
a deal there.

 

Thanks

 





--
 Jon Lewis, MCP :)   |  I route
 |  therefore you are
_ http://www.lewis.org/~jlewis/pgp for PGP public key_

Was wrong Re: Did IPv6 between HE and Google ever get resolved?

[David Hubbard]

Oops, I was corrected that HE doesn’t have IPv6 issues with Google, not sure 
why I had that in my head.  Cogent certainly does but something had me thinking 
there’s another big name that has the same problem.

David

From: NANOG  on behalf of David Hubbard 

Date: Thursday, March 28, 2019 at 12:40 PM
To: NANOG List 
Subject: Did IPv6 between HE and Google ever get resolved?

Hey all, I’ve been having bad luck searching around, but did IPv6 transit 
between HE and google ever get resolved?  Ironically, I can now get to them 
cheaply from a location we currently have equipment that has been Cogent-only, 
so if it fixes the IPv6 issue I’d like to make the move.  Anyone peer with HE 
in general and want to share their experience offlist?  With the price, if 
they’re a good option, I’d consider rolling them in to other locations where we 
have redundancy already, so the v6 isn’t as big a deal there.

Thanks

Did IPv6 between HE and Google ever get resolved?

[David Hubbard]

Hey all, I’ve been having bad luck searching around, but did IPv6 transit 
between HE and google ever get resolved?  Ironically, I can now get to them 
cheaply from a location we currently have equipment that has been Cogent-only, 
so if it fixes the IPv6 issue I’d like to make the move.  Anyone peer with HE 
in general and want to share their experience offlist?  With the price, if 
they’re a good option, I’d consider rolling them in to other locations where we 
have redundancy already, so the v6 isn’t as big a deal there.

Thanks

Re: Banned by Akamai (or some websites hosted with Akamai)

[Jared Mauch]

On Thu, Mar 28, 2019 at 06:17:25AM -0500, Mike Hammett wrote:
> Hopefully Jared can fix it. Owen's description matches up very well with my 
> experiences in trying to fix similar problems at Akamai. 

Don't worry, I can't access my car owners insurance website
from the country i'm in as well due to a similar WAF config on another
CDN.

I've replied to both people that posted to the list with some
further details.  Don't hesitate to reach out if you're not getting
a response or have questions about your experiences with akamai.

We are here and will do our best to fix things, but also
similar to my car insurance folks who don't want me to have access
from this country, keep in mind our customers may also have configured
policy to block certain clients or behaviors.

I can reach out to the account teams to have them confirm
with customer the config is right if it seems odd.

- Jared

-- 
Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
clue++;  | http://puck.nether.net/~jared/  My statements are only mine.

Re: Advertisement of Equinix Chicago IX Subnet

[Job Snijders]

On Wed, Mar 27, 2019 at 09:36:20PM +, Graham Johnston wrote:
> This afternoon at around 12:17 central time today we began learning
> the subnet for the Equinix IX in Chicago via a transit provider; we
> are on the IX as well. The subnet in question is 208.115.136.0/23.
> Using stat.ripe.net I can see that this subnet is also being learned
> by others, see the snip below. On our network this caused a nasty
> routing loop until we figured out what was wrong. My current best
> understanding is that because the route was learned via eBGP it
> trumped the OSPF learned route. As soon as I filtered the
> advertisement from my transit provider everything returned to normal.
> What am I doing that isn’t best practices that would have prevented
> this?

There is two pieces to help prevent this type of failure:

1/ Equinix should have created a RPKI ROA for 208.115.136.0/23, with an
   Origin ASN of 0 or one of their own ASNs, and a Max Length of 23.

2/ You should implement RPKI based BGP Origin Validation in your network
   and honor those ROAs.

Kind regards,

Job

Re: Advertisement of Equinix Chicago IX Subnet

[Job Snijders]

On Thu, Mar 28, 2019 at 02:59:43PM +0100, Niels Bakker wrote:
> * christopher.morrell.na...@gmail.com (Christopher Morrell) [Thu 28 Mar 2019, 
> 14:35 CET]:
> > I've been bit by this in the past at two different exchanges. I too
> > have a policy applied to deny IXP LANs from upstreams and peers. It
> > would be nice if there was a list of all IXP LANs somewhere that we
> > could generically add to all upstream and peers.
> 
> I like Nick Hilliard's posted solution much better than creating
> static bogon lists that people will eventually forget about.

IXPs can use RPKI ROAs to signal to the world what their intentions are!
IXPs could either create a ROA with an Origin ASN of '0' to suggest to
the world that the peering lan prefix should never be visible in the
DFZ, or they can specify their own services ASN and simply not announce
the prefix. In either case IXPs should carefully specify the Max Length
value to be the same as the Prefix Length value of the peering lan
prefix.

Kind regards,

Job

Re: Advertisement of Equinix Chicago IX Subnet

[Niels Bakker]

* christopher.morrell.na...@gmail.com (Christopher Morrell) [Thu 28 Mar 2019, 
14:35 CET]:
I've been bit by this in the past at two different exchanges.  I too 
have a policy applied to deny IXP LANs from upstreams and peers.  It 
would be nice if there was a list of all IXP LANs somewhere that we 
could generically add to all upstream and peers.


I like Nick Hilliard's posted solution much better than creating 
static bogon lists that people will eventually forget about.



-- Niels.

Re: Advertisement of Equinix Chicago IX Subnet

[Christopher Morrell]

I've been bit by this in the past at two different exchanges.  I too have a
policy applied to deny IXP LANs from upstreams and peers.  It would be nice
if there was a list of all IXP LANs somewhere that we could generically add
to all upstream and peers.


On Thu, Mar 28, 2019 at 9:11 AM Eric Dugas  wrote:

> I have a policy applied to my upstreams and peers to deny the IXP's LANs
> were connected to. I don't think of any reason to learn these routes from
> someone else's network.
>
> On Wed, Mar 27, 2019 at 7:44 PM Cummings, Chris 
> wrote:
>
>> Not too sure about your topology, but I’ve had something similar bite me,
>> so we typically put a prefix list inbound to deny receiving our internal
>> prefixes from our peers. This probably doesn’t work as well if your network
>> is less “eyeballish” than ours, however.
>>
>> /chris
>>
>>
>>
>> On Wed, Mar 27, 2019 at 4:37 PM -0500, "Graham Johnston" <
>> johnst...@westmancom.com> wrote:
>>
>> This afternoon at around 12:17 central time today we began learning the
>>> subnet for the Equinix IX in Chicago via a transit provider; we are on the
>>> IX as well. The subnet in question is 208.115.136.0/23. Using
>>> stat.ripe.net
>>> 
>>> I can see that this subnet is also being learned by others, see the snip
>>> below. On our network this caused a nasty routing loop until we figured out
>>> what was wrong. My current best understanding is that because the route was
>>> learned via eBGP it trumped the OSPF learned route. As soon as I filtered
>>> the advertisement from my transit provider everything returned to normal.
>>> What am I doing that isn’t best practices that would have prevented this?
>>>
>>>
>>>
>>> Thanks,
>>>
>>> graham
>>>
>>>
>>>
>>>
>>>
>>> RIPE Info
>>>
>>> *1* RRCs see *1* peers announcing *208.115.136.0/23
>>> * originated by *AS32703*
>>> 
>>>
>>> · ▼RRC00 in *Amsterdam, Netherlands* sees *1* ASN orginating 
>>> *208.115.136.0/23
>>> *.AS32703
>>>
>>> o▼*AS32703
>>> *
>>>  is
>>> seen as the origin by *1* peer.192.102.254.1
>>>
>>> §  ▼*192.102.254.1
>>> *
>>>  is
>>> announcing route *AS395152*
>>> 
>>>  *AS63297*
>>> 
>>>  *AS6327*
>>> 
>>>  *AS36280*
>>> 
>>> *AS32703*
>>> 
>>> .
>>>
>>> §  Origin: IGP
>>>
>>> §  Next Hop: 192.102.254.1
>>>
>>> §  Peer: 192.102.254.1
>>>
>>> §  Community: 63297:1000
>>>
>>> §  AS Path: 395152 63297 6327 36280 32703
>>>
>>> §  Last Updated: 2019-03-27T17:17:19
>>>
>>>
>>>
>>>
>>>
>>> Route-views
>>>
>>> route-views.chicago.routeviews.org
>>> >
>>> show ip bgp 208.115.136.0
>>>
>>> BGP routing table entry for 208.115.136.0/23
>>>
>>> Paths: (1 available, best #1, table Default-IP-Routing-Table)
>>>
>>>   Not advertised to any peer
>>>
>>>   32709 32703
>>>
>>> 208.115.136.134 from 208.115.136.134 (63.134.128.248)
>>>
>>>   Origin IGP, localpref 100, valid, external, best
>>>
>>>   AddPath ID: RX 0, TX 64414249
>>>
>>>   Last update: Wed Mar 27 17:16:09 2019
>>>
>>

Re: Advertisement of Equinix Chicago IX Subnet

[Eric Dugas]

I have a policy applied to my upstreams and peers to deny the IXP's LANs
were connected to. I don't think of any reason to learn these routes from
someone else's network.

On Wed, Mar 27, 2019 at 7:44 PM Cummings, Chris  wrote:

> Not too sure about your topology, but I’ve had something similar bite me,
> so we typically put a prefix list inbound to deny receiving our internal
> prefixes from our peers. This probably doesn’t work as well if your network
> is less “eyeballish” than ours, however.
>
> /chris
>
>
>
> On Wed, Mar 27, 2019 at 4:37 PM -0500, "Graham Johnston" <
> johnst...@westmancom.com> wrote:
>
> This afternoon at around 12:17 central time today we began learning the
>> subnet for the Equinix IX in Chicago via a transit provider; we are on the
>> IX as well. The subnet in question is 208.115.136.0/23. Using
>> stat.ripe.net
>> 
>> I can see that this subnet is also being learned by others, see the snip
>> below. On our network this caused a nasty routing loop until we figured out
>> what was wrong. My current best understanding is that because the route was
>> learned via eBGP it trumped the OSPF learned route. As soon as I filtered
>> the advertisement from my transit provider everything returned to normal.
>> What am I doing that isn’t best practices that would have prevented this?
>>
>>
>>
>> Thanks,
>>
>> graham
>>
>>
>>
>>
>>
>> RIPE Info
>>
>> *1* RRCs see *1* peers announcing *208.115.136.0/23
>> * originated by *AS32703*
>> 
>>
>> · ▼RRC00 in *Amsterdam, Netherlands* sees *1* ASN orginating 
>> *208.115.136.0/23
>> *.AS32703
>>
>> o▼*AS32703
>> *
>>  is
>> seen as the origin by *1* peer.192.102.254.1
>>
>> §  ▼*192.102.254.1
>> *
>>  is
>> announcing route *AS395152*
>> 
>>  *AS63297*
>> 
>>  *AS6327*
>> 
>>  *AS36280*
>> 
>> *AS32703*
>> 
>> .
>>
>> §  Origin: IGP
>>
>> §  Next Hop: 192.102.254.1
>>
>> §  Peer: 192.102.254.1
>>
>> §  Community: 63297:1000
>>
>> §  AS Path: 395152 63297 6327 36280 32703
>>
>> §  Last Updated: 2019-03-27T17:17:19
>>
>>
>>
>>
>>
>> Route-views
>>
>> route-views.chicago.routeviews.org
>> >
>> show ip bgp 208.115.136.0
>>
>> BGP routing table entry for 208.115.136.0/23
>>
>> Paths: (1 available, best #1, table Default-IP-Routing-Table)
>>
>>   Not advertised to any peer
>>
>>   32709 32703
>>
>> 208.115.136.134 from 208.115.136.134 (63.134.128.248)
>>
>>   Origin IGP, localpref 100, valid, external, best
>>
>>   AddPath ID: RX 0, TX 64414249
>>
>>   Last update: Wed Mar 27 17:16:09 2019
>>
>

Re: Banned by Akamai (or some websites hosted with Akamai)

[Mike Hammett]

Hopefully Jared can fix it. Owen's description matches up very well with my 
experiences in trying to fix similar problems at Akamai. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Jared Mauch"  
To: "Owen DeLong"  
Cc: nanog@nanog.org 
Sent: Wednesday, March 27, 2019 12:25:32 PM 
Subject: Re: Banned by Akamai (or some websites hosted with Akamai) 

All companies have unique challenges in trying to mitigate abuse and serve 
customers well. 

Miao I’ll collect details from you in private to see if there is something that 
can be done. 

Sent from my iCar 

> On Mar 27, 2019, at 4:56 PM, Owen DeLong  wrote: 
> 
> Akamai will _NOT_ be helpful in this situation. 
> 
> They will tell you that it is their customers who set the policy for their 
> “Web Application Firewall”. 
> 
> In reality, Akamai’s customers set certain things on “autopilot” where Akamai 
> maintains a reputation database for various IP addresses and triggers actions 
> set by their customers without their customers direct knowledge or 
> intervention. 
> 
> Akamai’s process for dealing with this (or rather their refusal to create a 
> process for dealing with it) is a horrible disservice to the internet and to 
> their customers. 
> 
> I tried to push for changes to this process while I was there and had no 
> significant success. 
> 
> I’ve also been the victim of these practices after I was laid off by Akamai 
> (along with about 7% of their employees last year). 
> 
> Because of a variety of issues I’m not at liberty to elaborate, it isn’t an 
> easy problem for Akamai to solve, but as a company that prides itself on 
> tackling and solving difficult problems, they’ve certainly fallen short here. 
> 
> Owen 
> 
> 
>> On Mar 27, 2019, at 08:46 , Siyuan Miao  wrote: 
>> 
>> Hi, 
>> 
>> I got some complaints from customers and found out that all IP addresses 
>> announced in one of our ASN are banned by Akamai or some websites hosted 
>> with Akamai. 
>> 
>> I've tried to contact one of the website owners but didn't get any response. 
>> 
>> Could someone from Akamai contact me off-list? 
>> 
>> Regards, 
>> Siyuan Miao 

Re: residential/smb internet access in 2019 - help?

[Mike Hammett]

Variability will always happen with small businesses, but you're more likely to 
encounter someone that won't do nasty things to your bits through a local WISP 
as opposed to a national player. It's also more likely to be consistent versus 
the variability of a mobile service. 

WISPs have been going strong for years. 

Typically when a fixed wireless customer moves to mobile wireless, they move 
back within a couple months. 




Also, *most* people don't need more than 10 megs at home, so fixed providers 
that haven't upgraded to support faster speeds aren't really at a disadvantage 
when you look at how the connection is actually used. That becomes apparent 
once you switch. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Bryan Fields"  
To: "NANOG List"  
Sent: Wednesday, March 27, 2019 2:28:05 PM 
Subject: Re: residential/smb internet access in 2019 - help? 

On 3/27/19 7:50 AM, Mike Hammett wrote: 
> https://broadbandnow.com/Florida/Micanopy?zip=32667# 
> 
> You might want to try neighboring ZIP codes to see what other fixed 
> wireless providers might be convinced to expand. 
> 
> http://svic.net/wireless-broadband-north-florida/ 

You really want to weigh what wireless can offer as many of the local players 
doing wireless lack the depth of network knowledge and are completely ignorant 
of what it takes to run an RF network. I'd independently verify your circuits 
up-time if you decide to go with a wireless ISP. 

The other sad part is the PtMP wireless technology is likely slower than an 
LTE modem with external antenna. 

The WISP's had a great time circa 2005 or so, but now that the licensed 
players have surpassed what they can offer it's hard to justify the lower 
availability of the typical WISP vs. cost. 

-- 
Bryan Fields 

727-409-1194 - Voice 
http://bryanfields.net 

Nexus 9396 SNMP Issues

[Mike Hammett]

Does anyone else have issues with the 9396 sending out bum SNMP responses? 

Seemingly all DDM information for the optic modules return just a single digit. 
IE: 


[redacted]# show int eth 1/3 trans det 
Ethernet1/3 
transceiver is present 
type is 1000base-LH 
name is Fiberstore 
part number is SFP1G-LH-31 
revision is A0 
serial number is F16ACO17646 
nominal bitrate is 1300 MBit/sec 
Link length supported for 9/125um fiber is 10 km 
cisco id is 3 
cisco extended id number is 4 


SFP Detail Diagnostics Information (internal calibration) 
 
Current Alarms Warnings 
Measurement High Low High Low 
 
Temperature 40.72 C 100.00 C -50.00 C 85.00 C -40.00 C 
Voltage 3.35 V 3.79 V 2.80 V 3.46 V 3.13 V 
Current 15.89 mA 90.00 mA 0.00 mA 85.00 mA 0.00 mA 
Tx Power -6.05 dBm -1.50 dBm -10.50 dBm -3.00 dBm -9.03 dBm 
Rx Power -6.32 dBm -3.00 dBm -26.98 dBm -5.00 dBm -23.97 dBm 
Transmit Fault Count = 0 
 
Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning 



[redacted]:~$ /usr/bin/snmpget -v2c -c [redacted] 
.1.3.6.1.4.1.9.9.91.1.1.1.1.4.33533 .1.3.6.1.4.1.9.9.91.1.1.1.1.4.33534 
iso.3.6.1.4.1.9.9.91.1.1.1.1.4.33533 = INTEGER: -6 
iso.3.6.1.4.1.9.9.91.1.1.1.1.4.33534 = INTEGER: -6 


[redacted]:/opt/librenms# tcpdump host [redacted] 
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode 
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes 
11:13:33.360509 IP [redacted].49594 > [redacted].snmp: C="[redacted]" 
GetRequest(62) E:cisco.9.91.1.1.1.1.4.33533 
E:cisco.9.91.1.1.1.1.4.33534 
11:13:33.362093 IP [redacted].snmp > [redacted].49594: C="[redacted]" 
GetResponse(64) E:cisco.9.91.1.1.1.1.4.33533=-6 
E:cisco.9.91.1.1.1.1.4.33534=-6 
^C 
2 packets captured 
2 packets received by filter 
0 packets dropped by kernel 






Here I have a 3064 that reports just fine. 










[redacted]# show int eth 1/17 trans det 
Ethernet1/17 
transceiver is present 
type is 1000base-LH 
name is FiberStore 
part number is SFP1G-LX-31 
revision is A0 
serial number is D87B1487283 
nominal bitrate is 1300 MBit/sec 
Link length supported for 9/125um fiber is 10 km 
cisco id is 3 
cisco extended id number is 4 


SFP Detail Diagnostics Information (internal calibration) 
 
Current Alarms Warnings 
Measurement High Low High Low 
 
Temperature 33.38 C 100.00 C -50.00 C 85.00 C -40.00 C 
Voltage 3.33 V 3.79 V 2.80 V 3.46 V 3.13 V 
Current 19.60 mA 90.00 mA 0.00 mA 85.00 mA 0.00 mA 
Tx Power -6.10 dBm -1.50 dBm -10.50 dBm -3.00 dBm -9.03 dBm 
Rx Power -6.94 dBm 0.99 dBm -30.00 dBm -1.00 dBm -26.98 dBm 
Transmit Fault Count = 0 
 
Note: ++ high-alarm; + high-warning; -- low-alarm; - low-warning 


[redacted]:/opt/librenms# /usr/bin/snmpget -v2c -c [redacted] 
.1.3.6.1.4.1.9.9.91.1.1.1.1.4.300028173 .1.3.6.1.4.1.9.9.91.1.1.1.1.4.300028174 
iso.3.6.1.4.1.9.9.91.1.1.1.1.4.300028173 = INTEGER: -6968 
iso.3.6.1.4.1.9.9.91.1.1.1.1.4.300028174 = INTEGER: -6090 




[redacted]:/opt/librenms# tcpdump host [redacted] 
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode 
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes 
11:54:01.25 IP [redacted].36131 > [redacted].snmp: C="[redacted]" 
GetRequest(62) E:cisco.9.91.1.1.1.1.4.300028173 
E:cisco.9.91.1.1.1.1.4.300028174 
11:54:01.261027 IP [redacted].snmp > [redacted].36131: C="[redacted]" 
GetResponse(66) E:cisco.9.91.1.1.1.1.4.300028173=-6968 
E:cisco.9.91.1.1.1.1.4.300028174=-6090 
^C 
2 packets captured 
2 packets received by filter 
0 packets dropped by kernel 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com