Re: Google you have a problem.

[Christopher Morrow]

On Thu, Jan 31, 2019 at 8:23 PM Mark Andrews  wrote:

> It was 9:29 AM Feb 1 AEST when I reported this so yes it was FRIDAY.
>
>
:) it's very easy to rile you up.

Re: Google you have a problem.

[Mark Andrews]

It was 9:29 AM Feb 1 AEST when I reported this so yes it was FRIDAY.

> On 1 Feb 2019, at 3:15 pm, Christopher Morrow  wrote:
> 
> 
> 
> On Thu, Jan 31, 2019 at 4:34 PM Mark Andrews  wrote:
> George Michaelson forwarded me Google’s notice which said there would be a
> temporary outage along with a new home.  Time to update the code to use the
> new home.
> 
> 
> oh hey neato! :)
> Also, hey! this change didn't happen on a friday! w00t!
>  
> > On 1 Feb 2019, at 11:31 am, Christopher Morrow  
> > wrote:
> > 
> > just in case.. .this appears to be working now.
> > -chris
> > 
> > On Thu, Jan 31, 2019 at 2:30 PM Mark Andrews  wrote:
> > [beetle:~/git/bind9] marka% fetch -v https://www.google.com/jsapi
> > looking up www.google.com
> > connecting to www.google.com:443
> > SSL connection established using ECDHE-RSA-AES128-GCM-SHA256
> > Certificate subject: /C=US/ST=California/L=Mountain View/O=Google 
> > LLC/CN=www.google.com
> > Certificate issuer: /C=US/O=Google Trust Services/CN=Google Internet 
> > Authority G3
> > requesting https://www.google.com/jsapi
> > fetch: https://www.google.com/jsapi: Bad Gateway
> > [beetle:~/git/bind9] marka% curl -v https://www.google.com/jsapi
> > *   Trying 2607:f8b0:4007:80e::2004...
> > * TCP_NODELAY set
> > *   Trying 172.217.14.100...
> > * TCP_NODELAY set
> > * Connected to www.google.com (2607:f8b0:4007:80e::2004) port 443 (#0)
> > * ALPN, offering http/1.1
> > * Cipher selection: 
> > ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
> > * successfully set certificate verify locations:
> > *   CAfile: /opt/local/share/curl/curl-ca-bundle.crt
> >   CApath: none
> > * TLSv1.2 (OUT), TLS header, Certificate Status (22):
> > * TLSv1.2 (OUT), TLS handshake, Client hello (1):
> > * TLSv1.2 (IN), TLS handshake, Server hello (2):
> > * TLSv1.2 (IN), TLS handshake, Certificate (11):
> > * TLSv1.2 (IN), TLS handshake, Server key exchange (12):
> > * TLSv1.2 (IN), TLS handshake, Server finished (14):
> > * TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
> > * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
> > * TLSv1.2 (OUT), TLS handshake, Finished (20):
> > * TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
> > * TLSv1.2 (IN), TLS handshake, Finished (20):
> > * SSL connection using TLSv1.2 / ECDHE-ECDSA-AES128-GCM-SHA256
> > * ALPN, server accepted to use http/1.1
> > * Server certificate:
> > *  subject: C=US; ST=California; L=Mountain View; O=Google LLC; 
> > CN=www.google.com
> > *  start date: Jan 15 13:15:00 2019 GMT
> > *  expire date: Apr  9 13:15:00 2019 GMT
> > *  subjectAltName: host "www.google.com" matched cert's "www.google.com"
> > *  issuer: C=US; O=Google Trust Services; CN=Google Internet Authority G3
> > *  SSL certificate verify ok.
> > > GET /jsapi HTTP/1.1
> > > Host: www.google.com
> > > User-Agent: curl/7.63.0
> > > Accept: */*
> > > 
> > < HTTP/1.1 502 Bad Gateway
> > < Content-Length: 0
> > < Date: Thu, 31 Jan 2019 22:20:34 GMT
> > < Content-Type: text/html; charset=UTF-8
> > < Alt-Svc: quic=":443"; ma=2592000; v="44,43,39"
> > < 
> > * Connection #0 to host www.google.com left intact
> > [beetle:~/git/bind9] marka% curl -4 -v https://www.google.com/jsapi
> > *   Trying 172.217.14.100...
> > * TCP_NODELAY set
> > * Connected to www.google.com (172.217.14.100) port 443 (#0)
> > * ALPN, offering http/1.1
> > * Cipher selection: 
> > ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
> > * successfully set certificate verify locations:
> > *   CAfile: /opt/local/share/curl/curl-ca-bundle.crt
> >   CApath: none
> > * TLSv1.2 (OUT), TLS header, Certificate Status (22):
> > * TLSv1.2 (OUT), TLS handshake, Client hello (1):
> > * TLSv1.2 (IN), TLS handshake, Server hello (2):
> > * TLSv1.2 (IN), TLS handshake, Certificate (11):
> > * TLSv1.2 (IN), TLS handshake, Server key exchange (12):
> > * TLSv1.2 (IN), TLS handshake, Server finished (14):
> > * TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
> > * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
> > * TLSv1.2 (OUT), TLS handshake, Finished (20):
> > * TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
> > * TLSv1.2 (IN), TLS handshake, Finished (20):
> > * SSL connection using TLSv1.2 / ECDHE-ECDSA-AES128-GCM-SHA256
> > * ALPN, server accepted to use http/1.1
> > * Server certificate:
> > *  subject: C=US; ST=California; L=Mountain View; O=Google LLC; 
> > CN=www.google.com
> > *  start date: Jan 15 13:15:00 2019 GMT
> > *  expire date: Apr  9 13:15:00 2019 GMT
> > *  subjectAltName: host "www.google.com" matched cert's "www.google.com"
> > *  issuer: C=US; O=Google Trust Services; CN=Google Internet Authority G3
> > *  SSL certificate verify ok.
> > > GET /jsapi HTTP/1.1
> > > Host: www.google.com
> > > User-Agent: curl/7.63.0
> > > Accept: */*
> > > 
> > < HTTP/1.1 502 Bad Gateway
> > < Content-Length: 0
> > < Date: Thu, 31 Jan 2019 22:24:43 GMT
> > < Content-Type: text/html; charset=UTF-8
> > < Alt-Svc: quic=":443"; ma=2592000; v="44

Re: Google you have a problem.

[Christopher Morrow]

On Thu, Jan 31, 2019 at 4:34 PM Mark Andrews  wrote:

> George Michaelson forwarded me Google’s notice which said there would be a
> temporary outage along with a new home.  Time to update the code to use the
> new home.
>
>
oh hey neato! :)
Also, hey! this change didn't happen on a friday! w00t!


> > On 1 Feb 2019, at 11:31 am, Christopher Morrow 
> wrote:
> >
> > just in case.. .this appears to be working now.
> > -chris
> >
> > On Thu, Jan 31, 2019 at 2:30 PM Mark Andrews  wrote:
> > [beetle:~/git/bind9] marka% fetch -v https://www.google.com/jsapi
> > looking up www.google.com
> > connecting to www.google.com:443
> > SSL connection established using ECDHE-RSA-AES128-GCM-SHA256
> > Certificate subject: /C=US/ST=California/L=Mountain View/O=Google LLC/CN=
> www.google.com
> > Certificate issuer: /C=US/O=Google Trust Services/CN=Google Internet
> Authority G3
> > requesting https://www.google.com/jsapi
> > fetch: https://www.google.com/jsapi: Bad Gateway
> > [beetle:~/git/bind9] marka% curl -v https://www.google.com/jsapi
> > *   Trying 2607:f8b0:4007:80e::2004...
> > * TCP_NODELAY set
> > *   Trying 172.217.14.100...
> > * TCP_NODELAY set
> > * Connected to www.google.com (2607:f8b0:4007:80e::2004) port 443 (#0)
> > * ALPN, offering http/1.1
> > * Cipher selection:
> ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
> > * successfully set certificate verify locations:
> > *   CAfile: /opt/local/share/curl/curl-ca-bundle.crt
> >   CApath: none
> > * TLSv1.2 (OUT), TLS header, Certificate Status (22):
> > * TLSv1.2 (OUT), TLS handshake, Client hello (1):
> > * TLSv1.2 (IN), TLS handshake, Server hello (2):
> > * TLSv1.2 (IN), TLS handshake, Certificate (11):
> > * TLSv1.2 (IN), TLS handshake, Server key exchange (12):
> > * TLSv1.2 (IN), TLS handshake, Server finished (14):
> > * TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
> > * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
> > * TLSv1.2 (OUT), TLS handshake, Finished (20):
> > * TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
> > * TLSv1.2 (IN), TLS handshake, Finished (20):
> > * SSL connection using TLSv1.2 / ECDHE-ECDSA-AES128-GCM-SHA256
> > * ALPN, server accepted to use http/1.1
> > * Server certificate:
> > *  subject: C=US; ST=California; L=Mountain View; O=Google LLC; CN=
> www.google.com
> > *  start date: Jan 15 13:15:00 2019 GMT
> > *  expire date: Apr  9 13:15:00 2019 GMT
> > *  subjectAltName: host "www.google.com" matched cert's "www.google.com"
> > *  issuer: C=US; O=Google Trust Services; CN=Google Internet Authority G3
> > *  SSL certificate verify ok.
> > > GET /jsapi HTTP/1.1
> > > Host: www.google.com
> > > User-Agent: curl/7.63.0
> > > Accept: */*
> > >
> > < HTTP/1.1 502 Bad Gateway
> > < Content-Length: 0
> > < Date: Thu, 31 Jan 2019 22:20:34 GMT
> > < Content-Type: text/html; charset=UTF-8
> > < Alt-Svc: quic=":443"; ma=2592000; v="44,43,39"
> > <
> > * Connection #0 to host www.google.com left intact
> > [beetle:~/git/bind9] marka% curl -4 -v https://www.google.com/jsapi
> > *   Trying 172.217.14.100...
> > * TCP_NODELAY set
> > * Connected to www.google.com (172.217.14.100) port 443 (#0)
> > * ALPN, offering http/1.1
> > * Cipher selection:
> ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
> > * successfully set certificate verify locations:
> > *   CAfile: /opt/local/share/curl/curl-ca-bundle.crt
> >   CApath: none
> > * TLSv1.2 (OUT), TLS header, Certificate Status (22):
> > * TLSv1.2 (OUT), TLS handshake, Client hello (1):
> > * TLSv1.2 (IN), TLS handshake, Server hello (2):
> > * TLSv1.2 (IN), TLS handshake, Certificate (11):
> > * TLSv1.2 (IN), TLS handshake, Server key exchange (12):
> > * TLSv1.2 (IN), TLS handshake, Server finished (14):
> > * TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
> > * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
> > * TLSv1.2 (OUT), TLS handshake, Finished (20):
> > * TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
> > * TLSv1.2 (IN), TLS handshake, Finished (20):
> > * SSL connection using TLSv1.2 / ECDHE-ECDSA-AES128-GCM-SHA256
> > * ALPN, server accepted to use http/1.1
> > * Server certificate:
> > *  subject: C=US; ST=California; L=Mountain View; O=Google LLC; CN=
> www.google.com
> > *  start date: Jan 15 13:15:00 2019 GMT
> > *  expire date: Apr  9 13:15:00 2019 GMT
> > *  subjectAltName: host "www.google.com" matched cert's "www.google.com"
> > *  issuer: C=US; O=Google Trust Services; CN=Google Internet Authority G3
> > *  SSL certificate verify ok.
> > > GET /jsapi HTTP/1.1
> > > Host: www.google.com
> > > User-Agent: curl/7.63.0
> > > Accept: */*
> > >
> > < HTTP/1.1 502 Bad Gateway
> > < Content-Length: 0
> > < Date: Thu, 31 Jan 2019 22:24:43 GMT
> > < Content-Type: text/html; charset=UTF-8
> > < Alt-Svc: quic=":443"; ma=2592000; v="44,43,39"
> > <
> > * Connection #0 to host www.google.com left intact
> > [beetle:~/git/bind9] marka% traceroute 172.217.14.100
> > traceroute to 172.217.14.100 (172.21

RE: RTBH no_export

[Michel Py]

> Alejandro Acosta wrote :
> One more thing, RFC7999 has category Informational

Point well taken.
A good thing, IMHO. If I remember correctly, I once opposed this text; not 
because it was a bad idea (standardizing is sometimes a good idea) but because 
I found it imprecise enough that it was not achieveing any goal and blurred the 
definition of what is a RTBH.

Michel.

TSI Disclaimer:  This message and any files or text attached to it are intended 
only for the recipients named above and contain information that may be 
confidential or privileged. If you are not the intended recipient, you must not 
forward, copy, use or otherwise disclose this communication or the information 
contained herein. In the event you have received this message in error, please 
notify the sender immediately by replying to this message, and then delete all 
copies of it from your system. Thank you!...

Re: Calling LinkedIn, Amazon and Akamai @ DE-CIX NY

[Niels Bakker]

* br...@shout.net (Bryan Holloway) [Fri 01 Feb 2019, 02:00 CET]:

What do IXes do (or can do) to enforce the completion of a renumbering?


Renumbering is not complicated, but it's a lot of work to do it well.

- Set a realistic time schedule well in advance.  Keep in mind that 
some have change control limitations that differ from what you may be 
used to in the ISP industry.  Talk about it in your regular membership 
meeting prior to the renumbering at the very least, and send mails to 
your general membership mailing lists.


- Make sure your contacts for all connected ISPs are up to date.  
Generate unique emails per connected party and require them to 
acknowledge.  If they don't, start making phone calls.


- Allow for an overlap of old and new IP space to avoid outages.  Don't 
set a flag day since not everybody will be able to make that due to 
timezones and aforementioned change control, but pick a week.  Give 
instructions on how to have duplicated BGP sessions to avoid outages 
while allowing all connected parties to migrate at their speed.


- Consider temporarily duplicating your route server infrastructure.

- Keep a very close eye on your arpwatch and other monitoring in case 
people make typos - and people will make typos.


- Be ready to move ports to a quarantine VLAN when they haven't 
renumbered in time, despite those previously mentioned personal 
communications, so you can reuse the IP address space elsewhere.


HTH,


-- Niels.

Re: RTBH no_export

[Alejandro Acosta]

One more thing, RFC7999 has category Informational

El 31/1/19 a las 16:21, Theodore Baschak escribió:
>
>> On Jan 31, 2019, at 1:28 PM, Roel Parijs > > wrote:
>>
>> For our BGP customers the problem is more complex. Our BGP customers
>> can send us the RTBH community, and we will drop the traffic at our
>> borders. Since we're only running a small network, we don't have the
>> capacity to deal with large attacks. If we would be able to forward
>> (and maybe alter it) this RTBH community towards our upstream
>> providers, the impact on our network would be limited. However, the
>> RFC states that an announcement tagged with the blackhole community
>> should get the no_advertise or no_export community.
>>
>> What is your opinion on this ?
>>
>
> In RFC7999 section 3.2 the first paragraph talks about what you're
> mentioning, NO_EXPORT and/or NO_ADVERTISE. It uses the word SHOULD.
> SHOULD has special meaning in RFCs, its not MUST. Its also not MAY.
> RFC2119 talks about the way these words should be interpreted. 
>
> In the next paragraph it says that extreme caution should be used when
> "purposefully propagating IP prefixes tagged with the BLACKHOLE
> community outside the local routing domain, unless policy explicitly
> aims at doing just that."
>
> So if your local routing policy is to propagate those blackholes on to
> your upstreams (and its mutually agreed and they're configured to
> accept them), then it can be done. Nothing technical in the RFC
> stopping that. 
>
> Theo
>

Re: Calling LinkedIn, Amazon and Akamai @ DE-CIX NY

[Bryan Holloway]

What do IXes do (or can do) to enforce the completion of a renumbering?

For example, Chicago Equinix announced a renumbering beginning of 2018, 
and while 99% of our peers have renumbered, we still have an albeit 
small handful who have not. (I will not name names.)


That situation didn't get nearly the "media attention" that DE-CIX New 
York did, if any.


The "end of migration period" was May 1st 2018; here we are.

What's the next step, if any?


On 1/30/19 4:05 PM, Thomas King wrote:

Hi all,

Thanks for your support! This helps us getting all peers on the new IPv4 
space.


Our looking glass shows which peers already have changed the IP settings 
(see section “BGP session established”) and which peers are still 
working on it (see section “BGP sessions down”):


https://lg.de-cix.net/routeservers/rs1_nyc_ipv4#sessions-up

Re: Google you have a problem.

[Mark Andrews]

George Michaelson forwarded me Google’s notice which said there would be a
temporary outage along with a new home.  Time to update the code to use the
new home.

> On 1 Feb 2019, at 11:31 am, Christopher Morrow  
> wrote:
> 
> just in case.. .this appears to be working now.
> -chris
> 
> On Thu, Jan 31, 2019 at 2:30 PM Mark Andrews  wrote:
> [beetle:~/git/bind9] marka% fetch -v https://www.google.com/jsapi
> looking up www.google.com
> connecting to www.google.com:443
> SSL connection established using ECDHE-RSA-AES128-GCM-SHA256
> Certificate subject: /C=US/ST=California/L=Mountain View/O=Google 
> LLC/CN=www.google.com
> Certificate issuer: /C=US/O=Google Trust Services/CN=Google Internet 
> Authority G3
> requesting https://www.google.com/jsapi
> fetch: https://www.google.com/jsapi: Bad Gateway
> [beetle:~/git/bind9] marka% curl -v https://www.google.com/jsapi
> *   Trying 2607:f8b0:4007:80e::2004...
> * TCP_NODELAY set
> *   Trying 172.217.14.100...
> * TCP_NODELAY set
> * Connected to www.google.com (2607:f8b0:4007:80e::2004) port 443 (#0)
> * ALPN, offering http/1.1
> * Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
> * successfully set certificate verify locations:
> *   CAfile: /opt/local/share/curl/curl-ca-bundle.crt
>   CApath: none
> * TLSv1.2 (OUT), TLS header, Certificate Status (22):
> * TLSv1.2 (OUT), TLS handshake, Client hello (1):
> * TLSv1.2 (IN), TLS handshake, Server hello (2):
> * TLSv1.2 (IN), TLS handshake, Certificate (11):
> * TLSv1.2 (IN), TLS handshake, Server key exchange (12):
> * TLSv1.2 (IN), TLS handshake, Server finished (14):
> * TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
> * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
> * TLSv1.2 (OUT), TLS handshake, Finished (20):
> * TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
> * TLSv1.2 (IN), TLS handshake, Finished (20):
> * SSL connection using TLSv1.2 / ECDHE-ECDSA-AES128-GCM-SHA256
> * ALPN, server accepted to use http/1.1
> * Server certificate:
> *  subject: C=US; ST=California; L=Mountain View; O=Google LLC; 
> CN=www.google.com
> *  start date: Jan 15 13:15:00 2019 GMT
> *  expire date: Apr  9 13:15:00 2019 GMT
> *  subjectAltName: host "www.google.com" matched cert's "www.google.com"
> *  issuer: C=US; O=Google Trust Services; CN=Google Internet Authority G3
> *  SSL certificate verify ok.
> > GET /jsapi HTTP/1.1
> > Host: www.google.com
> > User-Agent: curl/7.63.0
> > Accept: */*
> > 
> < HTTP/1.1 502 Bad Gateway
> < Content-Length: 0
> < Date: Thu, 31 Jan 2019 22:20:34 GMT
> < Content-Type: text/html; charset=UTF-8
> < Alt-Svc: quic=":443"; ma=2592000; v="44,43,39"
> < 
> * Connection #0 to host www.google.com left intact
> [beetle:~/git/bind9] marka% curl -4 -v https://www.google.com/jsapi
> *   Trying 172.217.14.100...
> * TCP_NODELAY set
> * Connected to www.google.com (172.217.14.100) port 443 (#0)
> * ALPN, offering http/1.1
> * Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
> * successfully set certificate verify locations:
> *   CAfile: /opt/local/share/curl/curl-ca-bundle.crt
>   CApath: none
> * TLSv1.2 (OUT), TLS header, Certificate Status (22):
> * TLSv1.2 (OUT), TLS handshake, Client hello (1):
> * TLSv1.2 (IN), TLS handshake, Server hello (2):
> * TLSv1.2 (IN), TLS handshake, Certificate (11):
> * TLSv1.2 (IN), TLS handshake, Server key exchange (12):
> * TLSv1.2 (IN), TLS handshake, Server finished (14):
> * TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
> * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
> * TLSv1.2 (OUT), TLS handshake, Finished (20):
> * TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
> * TLSv1.2 (IN), TLS handshake, Finished (20):
> * SSL connection using TLSv1.2 / ECDHE-ECDSA-AES128-GCM-SHA256
> * ALPN, server accepted to use http/1.1
> * Server certificate:
> *  subject: C=US; ST=California; L=Mountain View; O=Google LLC; 
> CN=www.google.com
> *  start date: Jan 15 13:15:00 2019 GMT
> *  expire date: Apr  9 13:15:00 2019 GMT
> *  subjectAltName: host "www.google.com" matched cert's "www.google.com"
> *  issuer: C=US; O=Google Trust Services; CN=Google Internet Authority G3
> *  SSL certificate verify ok.
> > GET /jsapi HTTP/1.1
> > Host: www.google.com
> > User-Agent: curl/7.63.0
> > Accept: */*
> > 
> < HTTP/1.1 502 Bad Gateway
> < Content-Length: 0
> < Date: Thu, 31 Jan 2019 22:24:43 GMT
> < Content-Type: text/html; charset=UTF-8
> < Alt-Svc: quic=":443"; ma=2592000; v="44,43,39"
> < 
> * Connection #0 to host www.google.com left intact
> [beetle:~/git/bind9] marka% traceroute 172.217.14.100
> traceroute to 172.217.14.100 (172.217.14.100), 64 hops max, 52 byte packets
>  1  172.30.42.65 (172.30.42.65)  1.340 ms  3.278 ms  1.673 ms
>  2  * * *
>  3  * * *
>  4  59.154.18.232 (59.154.18.232)  13.710 ms
> hu0-5-0-0.22btpr01.optus.net.au (59.154.18.234)  24.311 ms  13.662 ms
>  5  72.14.219.128 (72.14.219.128)  12.445 ms  43.781 ms  10.760 ms
>  6  108.170.

Re: Google you have a problem.

[Christopher Morrow]

just in case.. .this appears to be working now.
-chris

On Thu, Jan 31, 2019 at 2:30 PM Mark Andrews  wrote:

> [beetle:~/git/bind9] marka% fetch -v https://www.google.com/jsapi
> looking up www.google.com
> connecting to www.google.com:443
> SSL connection established using ECDHE-RSA-AES128-GCM-SHA256
> Certificate subject: /C=US/ST=California/L=Mountain View/O=Google LLC/CN=
> www.google.com
> Certificate issuer: /C=US/O=Google Trust Services/CN=Google Internet
> Authority G3
> requesting https://www.google.com/jsapi
> fetch: https://www.google.com/jsapi: Bad Gateway
> [beetle:~/git/bind9] marka% curl -v https://www.google.com/jsapi
> *   Trying 2607:f8b0:4007:80e::2004...
> * TCP_NODELAY set
> *   Trying 172.217.14.100...
> * TCP_NODELAY set
> * Connected to www.google.com (2607:f8b0:4007:80e::2004) port 443 (#0)
> * ALPN, offering http/1.1
> * Cipher selection:
> ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
> * successfully set certificate verify locations:
> *   CAfile: /opt/local/share/curl/curl-ca-bundle.crt
>   CApath: none
> * TLSv1.2 (OUT), TLS header, Certificate Status (22):
> * TLSv1.2 (OUT), TLS handshake, Client hello (1):
> * TLSv1.2 (IN), TLS handshake, Server hello (2):
> * TLSv1.2 (IN), TLS handshake, Certificate (11):
> * TLSv1.2 (IN), TLS handshake, Server key exchange (12):
> * TLSv1.2 (IN), TLS handshake, Server finished (14):
> * TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
> * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
> * TLSv1.2 (OUT), TLS handshake, Finished (20):
> * TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
> * TLSv1.2 (IN), TLS handshake, Finished (20):
> * SSL connection using TLSv1.2 / ECDHE-ECDSA-AES128-GCM-SHA256
> * ALPN, server accepted to use http/1.1
> * Server certificate:
> *  subject: C=US; ST=California; L=Mountain View; O=Google LLC; CN=
> www.google.com
> *  start date: Jan 15 13:15:00 2019 GMT
> *  expire date: Apr  9 13:15:00 2019 GMT
> *  subjectAltName: host "www.google.com" matched cert's "www.google.com"
> *  issuer: C=US; O=Google Trust Services; CN=Google Internet Authority G3
> *  SSL certificate verify ok.
> > GET /jsapi HTTP/1.1
> > Host: www.google.com
> > User-Agent: curl/7.63.0
> > Accept: */*
> >
> < HTTP/1.1 502 Bad Gateway
> < Content-Length: 0
> < Date: Thu, 31 Jan 2019 22:20:34 GMT
> < Content-Type: text/html; charset=UTF-8
> < Alt-Svc: quic=":443"; ma=2592000; v="44,43,39"
> <
> * Connection #0 to host www.google.com left intact
> [beetle:~/git/bind9] marka% curl -4 -v https://www.google.com/jsapi
> *   Trying 172.217.14.100...
> * TCP_NODELAY set
> * Connected to www.google.com (172.217.14.100) port 443 (#0)
> * ALPN, offering http/1.1
> * Cipher selection:
> ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
> * successfully set certificate verify locations:
> *   CAfile: /opt/local/share/curl/curl-ca-bundle.crt
>   CApath: none
> * TLSv1.2 (OUT), TLS header, Certificate Status (22):
> * TLSv1.2 (OUT), TLS handshake, Client hello (1):
> * TLSv1.2 (IN), TLS handshake, Server hello (2):
> * TLSv1.2 (IN), TLS handshake, Certificate (11):
> * TLSv1.2 (IN), TLS handshake, Server key exchange (12):
> * TLSv1.2 (IN), TLS handshake, Server finished (14):
> * TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
> * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
> * TLSv1.2 (OUT), TLS handshake, Finished (20):
> * TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
> * TLSv1.2 (IN), TLS handshake, Finished (20):
> * SSL connection using TLSv1.2 / ECDHE-ECDSA-AES128-GCM-SHA256
> * ALPN, server accepted to use http/1.1
> * Server certificate:
> *  subject: C=US; ST=California; L=Mountain View; O=Google LLC; CN=
> www.google.com
> *  start date: Jan 15 13:15:00 2019 GMT
> *  expire date: Apr  9 13:15:00 2019 GMT
> *  subjectAltName: host "www.google.com" matched cert's "www.google.com"
> *  issuer: C=US; O=Google Trust Services; CN=Google Internet Authority G3
> *  SSL certificate verify ok.
> > GET /jsapi HTTP/1.1
> > Host: www.google.com
> > User-Agent: curl/7.63.0
> > Accept: */*
> >
> < HTTP/1.1 502 Bad Gateway
> < Content-Length: 0
> < Date: Thu, 31 Jan 2019 22:24:43 GMT
> < Content-Type: text/html; charset=UTF-8
> < Alt-Svc: quic=":443"; ma=2592000; v="44,43,39"
> <
> * Connection #0 to host www.google.com left intact
> [beetle:~/git/bind9] marka% traceroute 172.217.14.100
> traceroute to 172.217.14.100 (172.217.14.100), 64 hops max, 52 byte packets
>  1  172.30.42.65 (172.30.42.65)  1.340 ms  3.278 ms  1.673 ms
>  2  * * *
>  3  * * *
>  4  59.154.18.232 (59.154.18.232)  13.710 ms
> hu0-5-0-0.22btpr01.optus.net.au (59.154.18.234)  24.311 ms  13.662 ms
>  5  72.14.219.128 (72.14.219.128)  12.445 ms  43.781 ms  10.760 ms
>  6  108.170.247.74 (108.170.247.74)  12.743 ms  12.797 ms
> 108.170.247.59 (108.170.247.59)  14.167 ms
>  7  216.239.50.172 (216.239.50.172)  159.864 ms  162.324 ms  157.546 ms
>  8  108.170.230.130 (108.170.230.130)  147.234 ms  151.094 ms 

Google you have a problem.

[Mark Andrews]

[beetle:~/git/bind9] marka% fetch -v https://www.google.com/jsapi
looking up www.google.com
connecting to www.google.com:443
SSL connection established using ECDHE-RSA-AES128-GCM-SHA256
Certificate subject: /C=US/ST=California/L=Mountain View/O=Google 
LLC/CN=www.google.com
Certificate issuer: /C=US/O=Google Trust Services/CN=Google Internet Authority 
G3
requesting https://www.google.com/jsapi
fetch: https://www.google.com/jsapi: Bad Gateway
[beetle:~/git/bind9] marka% curl -v https://www.google.com/jsapi
*   Trying 2607:f8b0:4007:80e::2004...
* TCP_NODELAY set
*   Trying 172.217.14.100...
* TCP_NODELAY set
* Connected to www.google.com (2607:f8b0:4007:80e::2004) port 443 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /opt/local/share/curl/curl-ca-bundle.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-ECDSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: C=US; ST=California; L=Mountain View; O=Google LLC; 
CN=www.google.com
*  start date: Jan 15 13:15:00 2019 GMT
*  expire date: Apr  9 13:15:00 2019 GMT
*  subjectAltName: host "www.google.com" matched cert's "www.google.com"
*  issuer: C=US; O=Google Trust Services; CN=Google Internet Authority G3
*  SSL certificate verify ok.
> GET /jsapi HTTP/1.1
> Host: www.google.com
> User-Agent: curl/7.63.0
> Accept: */*
> 
< HTTP/1.1 502 Bad Gateway
< Content-Length: 0
< Date: Thu, 31 Jan 2019 22:20:34 GMT
< Content-Type: text/html; charset=UTF-8
< Alt-Svc: quic=":443"; ma=2592000; v="44,43,39"
< 
* Connection #0 to host www.google.com left intact
[beetle:~/git/bind9] marka% curl -4 -v https://www.google.com/jsapi
*   Trying 172.217.14.100...
* TCP_NODELAY set
* Connected to www.google.com (172.217.14.100) port 443 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /opt/local/share/curl/curl-ca-bundle.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-ECDSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: C=US; ST=California; L=Mountain View; O=Google LLC; 
CN=www.google.com
*  start date: Jan 15 13:15:00 2019 GMT
*  expire date: Apr  9 13:15:00 2019 GMT
*  subjectAltName: host "www.google.com" matched cert's "www.google.com"
*  issuer: C=US; O=Google Trust Services; CN=Google Internet Authority G3
*  SSL certificate verify ok.
> GET /jsapi HTTP/1.1
> Host: www.google.com
> User-Agent: curl/7.63.0
> Accept: */*
> 
< HTTP/1.1 502 Bad Gateway
< Content-Length: 0
< Date: Thu, 31 Jan 2019 22:24:43 GMT
< Content-Type: text/html; charset=UTF-8
< Alt-Svc: quic=":443"; ma=2592000; v="44,43,39"
< 
* Connection #0 to host www.google.com left intact
[beetle:~/git/bind9] marka% traceroute 172.217.14.100
traceroute to 172.217.14.100 (172.217.14.100), 64 hops max, 52 byte packets
 1  172.30.42.65 (172.30.42.65)  1.340 ms  3.278 ms  1.673 ms
 2  * * *
 3  * * *
 4  59.154.18.232 (59.154.18.232)  13.710 ms
hu0-5-0-0.22btpr01.optus.net.au (59.154.18.234)  24.311 ms  13.662 ms
 5  72.14.219.128 (72.14.219.128)  12.445 ms  43.781 ms  10.760 ms
 6  108.170.247.74 (108.170.247.74)  12.743 ms  12.797 ms
108.170.247.59 (108.170.247.59)  14.167 ms
 7  216.239.50.172 (216.239.50.172)  159.864 ms  162.324 ms  157.546 ms
 8  108.170.230.130 (108.170.230.130)  147.234 ms  151.094 ms  152.088 ms
 9  108.170.247.129 (108.170.247.129)  152.073 ms  149.194 ms  149.055 ms
10  108.170.230.137 (108.170.230.137)  149.697 ms  150.042 ms
74.125.252.75 (74.125.252.75)  149.363 ms
11  lax31s01-in-f4.1e100.net (172.217.14.100)  158.438 ms  147.982 ms  149.593 
ms
[beetle:~/git/bind9] marka% 

-- 
Mark Andrews, 

Re: DNS Flag Day, Friday, Feb 1st, 2019

[Mark Andrews]

TIMEOUT is TIMEOUT.  The whole point of flag day is that you can’t
tell whether TIMEOUT is broken routing, packet loss or badly configured
firewall.  The DNS flag day site assumes the latter as does the old
resolver code.  We are moving to a state where resolvers assume the
former.

You get a report with Red or Orange.  Red reports have things which
need to be fixed NOW be it a routing issue or packet loss or a badly
configured firewall.

Mark

> On 1 Feb 2019, at 4:04 am, Mark Tinka  wrote:
> 
> 
> 
> On 31/Jan/19 18:32, James Stahr wrote:
> 
>> 
>> I think the advertised testing tool may be flawed as blocking TCP/53
>> is enough to receive a STOP from the dnsflagday web site.  It's been
>> my (possibly flawed) understanding that TCP/53 is an option for
>> clients but primarily it is a mechanism for the *server* to request
>> the client communicate using TCP/53 instead - this could be due to UDP
>> response size, anti-spoofing controls, etc...
> 
> On a similar note, we tested for all our self-hosted zones OK 2 weeks
> ago. However, in previous days, the summary result said "NO GOOD, THINGS
> WILL BE SLOW COME FLAG DAY". The detailed test showed IPv4 tested
> perfect, but IPv6 probes timed out.
> 
> The issue turned out to be an internal IPv6 routing/forwarding issue for
> our service within Century Link's (Level(3)'s) network. CL finally fixed
> that issue today and the flag day test tool is happy again.
> 
> Some of our partners/customers were concerned our name servers were not
> ready, based purely on the summary result of the test tool. Perhaps
> adding some intelligence about whether the issue is the name server or
> the transport may be helpful.
> 
> Mark.

-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742  INTERNET: ma...@isc.org

Re: DNS Flag Day, Friday, Feb 1st, 2019

[Mark Andrews]

> On 1 Feb 2019, at 3:32 am, James Stahr  wrote:
> 
> On 2019-01-31 08:15, Mark Andrews wrote:
> 
>> We actually have a hard time finding zones where all the servers are
>> broken enough to not work with servers that don’t fallback to plain
>> DNS on timeout.  We can find zones where some of the servers are like
>> that, but there is usually one or more servers that do respond
>> correctly.
>> Of the datasets I’ve looked at that 1 in 10,000 to 1 in 100,000 zones
>> will have problems with updated servers.
> 
> I think the advertised testing tool may be flawed as blocking TCP/53 is 
> enough to receive a STOP from the dnsflagday web site.  It's been my 
> (possibly flawed) understanding that TCP/53 is an option for clients but 
> primarily it is a mechanism for the *server* to request the client 
> communicate using TCP/53 instead - this could be due to UDP response size, 
> anti-spoofing controls, etc…

DNS is defined to be both TCP and UDP.  Blocking TCP has no real security 
benefit and comes from the MYTH that TCP is ONLY used for zone transfers.  Way 
too many times people have blocked TCP then have those same servers return TC=1 
which say USE TCP as the answer is TOO LARGE TO FIT IN UDP.

Foot, Gun, Shoot.

If you have a DNSSEC zone then TCP is effectively MANDATORY as clients still 
need to be able to
get answers from behind a firewall that is enforcing a 512 byte limit.  If you 
are running a recursive
server TCP is effectively MANDATORY as you have no control over the RRset size. 
 Lots of referrals REQUIRE TCP as the GLUE records are not OPTIONAL in a 
referral.  Yes, BIND has been setting TC=1 on referrals for MANY years now when 
it is required, it should be setting it on many more. So if you want WORKING 
DNS you do not block TCP.  Other DNS vendors also set TC=1 on some referrals.  
This means if you are hosting third party content then TCP is effectively 
MANDATORY as you have no content control.

RFC 1123 said that DNS/TCP is a SHOULD where SHOULD is defined as

 *"SHOULD"

  This word or the adjective "RECOMMENDED" means that there
  may exist valid reasons in particular circumstances to
  ignore this item, but the full implications should be
  understood and the case carefully weighed before choosing
  a different course.

I’ve yet to see anyone who has TCP blocked actually know all the places in the 
DNS protocol where doing so will cause things to break.  None of them have done 
the necessary homework to actually exercise the SHOULD.  There are lots of 
lemmings when it comes to DNS practices.  All implementations of DNS are 
REQUIRED to support DNS/TCP.

The DNS flag day site flags servers without TCP as broken which they *are*.  
Whether it should be red or orange is a matter for debate.

A referral that in bigger than 512 bytes without involving DNSSEC.

[beetle:~/git/bind9] marka% dig example.net @a.root-servers.net

; <<>> DiG 9.13.1+hotspot+add-prefetch+marka <<>> example.net 
@a.root-servers.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 54567
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 13, ADDITIONAL: 27
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1472
;; QUESTION SECTION:
;example.net.   IN  A

;; AUTHORITY SECTION:
net.172800  IN  NS  a.gtld-servers.net.
net.172800  IN  NS  b.gtld-servers.net.
net.172800  IN  NS  c.gtld-servers.net.
net.172800  IN  NS  d.gtld-servers.net.
net.172800  IN  NS  e.gtld-servers.net.
net.172800  IN  NS  f.gtld-servers.net.
net.172800  IN  NS  g.gtld-servers.net.
net.172800  IN  NS  h.gtld-servers.net.
net.172800  IN  NS  i.gtld-servers.net.
net.172800  IN  NS  j.gtld-servers.net.
net.172800  IN  NS  k.gtld-servers.net.
net.172800  IN  NS  l.gtld-servers.net.
net.172800  IN  NS  m.gtld-servers.net.

;; ADDITIONAL SECTION:
a.gtld-servers.net. 172800  IN  A   192.5.6.30
b.gtld-servers.net. 172800  IN  A   192.33.14.30
c.gtld-servers.net. 172800  IN  A   192.26.92.30
d.gtld-servers.net. 172800  IN  A   192.31.80.30
e.gtld-servers.net. 172800  IN  A   192.12.94.30
f.gtld-servers.net. 172800  IN  A   192.35.51.30
g.gtld-servers.net. 172800  IN  A   192.42.93.30
h.gtld-servers.net. 172800  IN  A   192.54.112.30
i.gtld-servers.net. 172800  IN  A   192.43.172.30
j.gtld-servers.net. 172800  IN  A   192.48.79.30
k.gtld-servers.net. 172800  IN  A   

RE: RTBH no_export

[Michel Py]

> Roel Parijs wrote:
> To minimize the impact of DDoS, I have setup RTBH. For our own customers, we 
> can set the RTBH community ourselves towards our transit suppliers and
> this works well. For our BGP customers the problem is more complex. Our BGP 
> customers can send us the RTBH community, and we will drop the traffic
> at our borders. Since we're only running a small network, we don't have the 
> capacity to deal with large attacks. If we would be able to forward (and maybe
> alter it) this RTBH community towards our upstream providers, the impact on 
> our network would be limited. However, the RFC states that an announcement
> tagged with the blackhole community should get the no_advertise or no_export 
> community.

I think the RFC is flexible enough; it's more about what you have agreed with 
your upstream(s) in terms of what they will accept as blackholes routes.
Many upstreams will accept a destination-based blackhole if the prefix belongs 
to you, but accepting blackholes for other prefixes or accepting source-based 
blackholes requires a lot of trust. It's more a political issue than a 
technical one, as I see it.

Michel.

TSI Disclaimer:  This message and any files or text attached to it are intended 
only for the recipients named above and contain information that may be 
confidential or privileged. If you are not the intended recipient, you must not 
forward, copy, use or otherwise disclose this communication or the information 
contained herein. In the event you have received this message in error, please 
notify the sender immediately by replying to this message, and then delete all 
copies of it from your system. Thank you!...

Re: DNS Flag Day, Friday, Feb 1st, 2019

[Doug Barton]

On 2019-01-31 08:32, James Stahr wrote:


I think the advertised testing tool may be flawed as blocking TCP/53
is enough to receive a STOP from the dnsflagday web site.  It's been
my (possibly flawed) understanding that TCP/53 is an option for
clients but primarily it is a mechanism for the *server* to request
the client communicate using TCP/53 instead - this could be due to UDP
response size, anti-spoofing controls, etc...


That understanding is flawed, but understandable given how deeply 
ingrained this misinformation has become in the zeitgeist. Sections 4.2 
and 4.2.2 of 1035 clearly state that TCP is an expected channel, not 
optional.


This is relevant operationally for two reasons. First while most folks 
make an effort to keep answers under 512 bytes for response time 
reasons, you cannot guarantee that someone, somewhere in your org won't 
add a record that overflows. Also, you are guaranteed to overflow at 
some point once you roll out DNSSEC. I've even seen seemingly mundane 
things like SRV records overflow 512 bytes.


The other reason it's relevant operationally is that there are more and 
more experimental projects in development now that involve using TCP, 
even without the need for truncation, as a way to have more assurance 
that a response is not spoofed. There are also efforts under way to 
evaluate whether "pipelining" DNS requests to servers that you are 
already sending a lot of requests to is ultimately more efficient than 
UDP at high volumes.


So, like lack of EDNS compliance, lack of TCP "compliance" here is going 
to be a limiting factor for the growth of new features, at minimum; and 
could result in actual breakage.


hope this helps,

Doug

Re: RTBH no_export

[Theodore Baschak]

> On Jan 31, 2019, at 1:28 PM, Roel Parijs  wrote:
> 
> For our BGP customers the problem is more complex. Our BGP customers can send 
> us the RTBH community, and we will drop the traffic at our borders. Since 
> we're only running a small network, we don't have the capacity to deal with 
> large attacks. If we would be able to forward (and maybe alter it) this RTBH 
> community towards our upstream providers, the impact on our network would be 
> limited. However, the RFC states that an announcement tagged with the 
> blackhole community should get the no_advertise or no_export community.
> 
> What is your opinion on this ?
> 

In RFC7999 section 3.2 the first paragraph talks about what you're mentioning, 
NO_EXPORT and/or NO_ADVERTISE. It uses the word SHOULD. SHOULD has special 
meaning in RFCs, its not MUST. Its also not MAY. RFC2119 talks about the way 
these words should be interpreted. 

In the next paragraph it says that extreme caution should be used when 
"purposefully propagating IP prefixes tagged with the BLACKHOLE community 
outside the local routing domain, unless policy explicitly aims at doing just 
that."

So if your local routing policy is to propagate those blackholes on to your 
upstreams (and its mutually agreed and they're configured to accept them), then 
it can be done. Nothing technical in the RFC stopping that. 

Theo

Re: RTBH no_export

[Nick Hilliard]

Roel Parijs wrote on 31/01/2019 19:28:

What is your opinion on this ?


you should implement a different community for upstream blackholing. 
This should be stripped at your upstream links and replaced with the 
provider's RTBH community.  Your provider will then handle export 
restrictions as they see fit.


Nick

Re: RTBH no_export

[Łukasz Bromirski]

> On 31 Jan 2019, at 20:28, Roel Parijs  wrote:
> 
> Hello NANOG,
> 
> To minimize the impact of DDoS, I have setup RTBH.
> For our own customers, we can set the RTBH community ourselves towards our 
> transit suppliers and this works well.
> 
> For our BGP customers the problem is more complex. Our BGP customers can send 
> us the RTBH community, and we will drop the traffic at our borders. Since 
> we're only running a small network, we don't have the capacity to deal with 
> large attacks. If we would be able to forward (and maybe alter it) this RTBH 
> community towards our upstream providers, the impact on our network would be 
> limited. However, the RFC states that an announcement tagged with the 
> blackhole community should get the no_advertise or no_export community.
> 
> What is your opinion on this ?

Community agreed between you and your peer is one thing, the other
is community agreed with your upstreams. If in addition you own the
customer IP space, it’s even less of a problem. 

And… if you upstreams agree to signal RTBH with you, it’s added bonus
for them - they’re stopping the flood at their own edges thanks to you.

win-win-win-drop ;)

— 
./

RE: Effects of Cold Front on Internet Infrastructure - U.S. Midwest

[Colin Stanners (lists)]

Out here in Manitoba we use unheated/no-electricity OSP fiber patch panel 
pedestals in some locations, those work without issue down to the occasional 
-40. Note that that’s using all high-quality components.

 

For Fletcher’s case, it’s also possible that:

-there had been water intrusion in a splice case or cable on the way – but then 
that tends to cause complete failure, either on the first occasion or not long 
after, and not repeating temperature-dependent fade.

-there’s a bad fusion splice on the way whose characteristics are affected by 
temperature.

 

My first step in such a case would be to OTDR the line (renting an OTDR if we 
were a company that didn’t own one) to see approximately where the issue is and 
to get an idea what kind of issue it is – Fletcher, I guess that your company 
did not do so at that time?

 

 

From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Mel Beckman
Sent: Thursday, January 31, 2019 12:26 PM
To: Fletcher Kittredge 
Cc: North American Network Operators' Group 
Subject: Re: Effects of Cold Front on Internet Infrastructure - U.S. Midwest

 

Fletcher, 

 

I don’t think that’s true. I find no specs on fiber dB loss being a function of 
ambient temperature. I do find fiber optic application data sheets for extreme 
temperature applications of -500F and +500F (spacecraft). You’d think if 
temperature affected fiber transmission characteristics, they’d see it in space.

 

What you likely were seeing was connector loss, owing either to improper 
installation, incorrect materials, or unheated regen enclosures.

 

Insertion loss (IL) failures, for instance, in the cold are a direct result of 
cable termination component shrinkage. That’s why regen and patch enclosures 
need to be heated as well as cooled. 


All fiber termination components have stated temperature limits. As 
temperatures approach -40F, the thermoplastic components in a cable's breakout, 
jacketing, and fiber fanout sections shrink more than the optical glass. 
Ruggedized connectors help somewhat, but the rule is that you can’t let optical 
connectors and assemblies get really cold (or really hot).

 

A typical spec for a single-mode OSP connector is:

 

Operating -30C (-22F) to +60C (+140F)

 

The range for the corresponding Single Mode fiber is:

 

Operating -55C (-67F) to +70C (+158F)
Storage -60C (-76F) to +70C (+158F)
Installation -30C (-22F) to +50C (+122F)

All professional outside plant engineers know these requirements. So if you’re 
seeing failures, somebody is breaking a rule.

 

 -mel

 

 

On Jan 30, 2019, at 3:05 PM, Fletcher Kittredge mailto:fkitt...@gwi.net> > wrote:

 

 

Cold changes the transmission characteristics of fiber. At one point we were 
renting some old dark fiber from the local telephone company in northern Maine. 
When it would get below -15%-degree F the dB would get bad enough that the link 
using that fiber would stop working. The telephone company was selling us dark 
fiber because regulation required them to. They refused to give us another 
fiber nor inspect/repair. They took the position they were required to sell us 
fiber, not working fiber.

 

 

On Wed, Jan 30, 2019 at 11:41 AM Mark Tinka mailto:mark.ti...@seacom.mu> > wrote:

For anyone running IP networks in the Midwest, are you having to do anything 
special to keep your networks up?

For the data centres, is this cold front a chance to reduce air conditioning 
costs, or is it actually straining the infrastructure?

I'm curious, from a +27-degree C summer's day here in Johannesburg.

Mark.




 

-- 

Fletcher Kittredge
GWI
207-602-1134

www.gwi.net  

 

RTBH no_export

[Roel Parijs]

Hello NANOG,

To minimize the impact of DDoS, I have setup RTBH.
For our own customers, we can set the RTBH community ourselves towards our
transit suppliers and this works well.

For our BGP customers the problem is more complex. Our BGP customers can
send us the RTBH community, and we will drop the traffic at our borders.
Since we're only running a small network, we don't have the capacity to
deal with large attacks. If we would be able to forward (and maybe alter
it) this RTBH community towards our upstream providers, the impact on our
network would be limited. However, the RFC states that an announcement
tagged with the blackhole community should get the no_advertise or
no_export community.

What is your opinion on this ?

Thanks in advance
Roel

Re: Latency between Dallas and west coast

[JASON BOTHE via NANOG]

Hi Nathan

My current rtd from DA6 to SV1 is 38ms and from DRT Houston to LA1 is 30ms

Jason 

> On Feb 1, 2019, at 04:40, Nathanael Catangay Cariaga  
> wrote:
> 
> thank you all for the responses.  i guess i would have to discuss this with 
> our provider.
> 
>> On Fri, Feb 1, 2019, 1:39 AM Tom Beecher > NYC to LA is in the high 60ms range, so no, 200ms from Dallas to US west 
>> coast is not expected. 
>> 
>> 
>> 
>> 
>>> On Thu, Jan 31, 2019 at 12:14 PM Mark Tinka  wrote:
>>> 
>>> 
>>> On 31/Jan/19 18:53, Mike Hammett wrote:
>>> 
 It's 180 ms from Dallas to Djibouti, so no, that much latency to the west 
 coast of the US is not normal.
>>> 
>>> Or from Gaborone to Frankfurt, which is some 184ms.
>>> 
>>> Short of long re-route paths or congested, high packet loss links, I'd not 
>>> expect the latency between any 2 points in the U.S. to hit 200ms.
>>> 
>>> Mark.

Re: BGP Experiment

[Randy Bush]

> I suspect simple bugs are found by vendor, complex bugs are not
> economic to find.

the running internet is complex and has a horrifying number of special
cases compounded by kiddies being clever.  no one, independent of
resource requirements, could build a lab to the scale needed to test.

and then there is ewd's famous quote about testing.

randy

Re: Effects of Cold Front on Internet Infrastructure - U.S. Midwest

[Mel Beckman]

Fletcher,

I don’t think that’s true. I find no specs on fiber dB loss being a function of 
ambient temperature. I do find fiber optic application data sheets for extreme 
temperature applications of -500F and +500F (spacecraft). You’d think if 
temperature affected fiber transmission characteristics, they’d see it in space.

What you likely were seeing was connector loss, owing either to improper 
installation, incorrect materials, or unheated regen enclosures.

Insertion loss (IL) failures, for instance, in the cold are a direct result of 
cable termination component shrinkage. That’s why regen and patch enclosures 
need to be heated as well as cooled.

All fiber termination components have stated temperature limits. As 
temperatures approach -40F, the thermoplastic components in a cable's breakout, 
jacketing, and fiber fanout sections shrink more than the optical glass. 
Ruggedized connectors help somewhat, but the rule is that you can’t let optical 
connectors and assemblies get really cold (or really hot).

A typical spec for a single-mode OSP connector is:

Operating -30C (-22F) to +60C (+140F)

The range for the corresponding Single Mode fiber is:

Operating -55C (-67F) to +70C (+158F)
Storage -60C (-76F) to +70C (+158F)
Installation -30C (-22F) to +50C (+122F)

All professional outside plant engineers know these requirements. So if you’re 
seeing failures, somebody is breaking a rule.

 -mel


On Jan 30, 2019, at 3:05 PM, Fletcher Kittredge 
mailto:fkitt...@gwi.net>> wrote:


Cold changes the transmission characteristics of fiber. At one point we were 
renting some old dark fiber from the local telephone company in northern Maine. 
When it would get below -15%-degree F the dB would get bad enough that the link 
using that fiber would stop working. The telephone company was selling us dark 
fiber because regulation required them to. They refused to give us another 
fiber nor inspect/repair. They took the position they were required to sell us 
fiber, not working fiber.


On Wed, Jan 30, 2019 at 11:41 AM Mark Tinka 
mailto:mark.ti...@seacom.mu>> wrote:
For anyone running IP networks in the Midwest, are you having to do anything 
special to keep your networks up?

For the data centres, is this cold front a chance to reduce air conditioning 
costs, or is it actually straining the infrastructure?

I'm curious, from a +27-degree C summer's day here in Johannesburg.

Mark.


--
Fletcher Kittredge
GWI
207-602-1134
www.gwi.net

Re: Latency between Dallas and west coast

[Nathanael Catangay Cariaga]

thank you all for the responses.  i guess i would have to discuss this with
our provider.

On Fri, Feb 1, 2019, 1:39 AM Tom Beecher  NYC to LA is in the high 60ms range, so no, 200ms from Dallas to US west
> coast is not expected.
>
>
>
>
> On Thu, Jan 31, 2019 at 12:14 PM Mark Tinka  wrote:
>
>>
>>
>> On 31/Jan/19 18:53, Mike Hammett wrote:
>>
>> It's 180 ms from Dallas to Djibouti, so no, that much latency to the west
>> coast of the US is not normal.
>>
>>
>> Or from Gaborone to Frankfurt, which is some 184ms.
>>
>> Short of long re-route paths or congested, high packet loss links, I'd
>> not expect the latency between any 2 points in the U.S. to hit 200ms.
>>
>> Mark.
>>
>

Re: Latency between Dallas and west coast

[Tom Beecher]

NYC to LA is in the high 60ms range, so no, 200ms from Dallas to US west
coast is not expected.




On Thu, Jan 31, 2019 at 12:14 PM Mark Tinka  wrote:

>
>
> On 31/Jan/19 18:53, Mike Hammett wrote:
>
> It's 180 ms from Dallas to Djibouti, so no, that much latency to the west
> coast of the US is not normal.
>
>
> Or from Gaborone to Frankfurt, which is some 184ms.
>
> Short of long re-route paths or congested, high packet loss links, I'd not
> expect the latency between any 2 points in the U.S. to hit 200ms.
>
> Mark.
>

Re: Latency between Dallas and west coast

[Mark Tinka]

On 31/Jan/19 18:53, Mike Hammett wrote:

> It's 180 ms from Dallas to Djibouti, so no, that much latency to the
> west coast of the US is not normal.

Or from Gaborone to Frankfurt, which is some 184ms.

Short of long re-route paths or congested, high packet loss links, I'd
not expect the latency between any 2 points in the U.S. to hit 200ms.

Mark.

Re: DNS Flag Day, Friday, Feb 1st, 2019

[Jimmy Hess]

On Thu, Jan 31, 2019 at 10:33 AM James Stahr  wrote:
[snip]
> So is the tool right in saying that TCP/53 is a absolute requirement of
> ENDS0 support for "DNS Flag Day"?  If so, do we expect a dramatic
> increases in TCP/53 requests over UDP/53 queries?  Or is the tool flawed
[snip]

Their test tool will obviously alert on more error conditions than
what the Flag Day is
specifically about --   One or more of your DNS servers not responding
[OR] TCP/53 not
working are still broken configurations,   But  the brokenness is
unrelated to what the flag
day is about -  In the first case,  better safe than sorry, I suppose:
 Inability to complete
one or more of the tests because of brokenness definitely means that
things are broken.

TCP/53 is a fairly strong requirement,  except if you are supporting
an authoritative-only
server with  no record labels that could result in a >512byte
response, plus no DNSSEC-secured zones,
and even then the AXFR protocol for replication to secondary servers
calls for TCP.

EDNS support is not required.   Authoritative servers that don't support EDNS
and are also compliant with the original DNS standard continue to work
after the workarounds are removed.

The relevant standard does not allow for silently ignoring requests
that contain the EDNS option;
patching the bug in a broken server does not necessarily entail the
larger task of adding EDNS support
-- achieving consistence compliance with either the DNS standard
before EDNS, or the DNS standard after
EDNS, is the requirement.

There are two ways for a DNS server to relay the DNS responses larger
than 512 bytes
1. The server replies with a truncated message with the truncate bit
set, and the client connects
to you over TCP to make the DNS request,OR   The client provided
the EDNS option with a larger packet size,
and you support that, so you send a large UDP reply instead.

A DNS server must support the first of these methods  (The second is
preferable but optional,  and support
for the first method over TCP is mandatory)  if you could ever be
required to handle
a DNS message whose reply is larger than 512 bytes:

All  recursive resolvers fall into this category, and with DNSSEC +
IPv6,   many common queries
will result in an answer longer than the original 512 byte limit of UDP.

--
-JH

Re: DNS Flag Day, Friday, Feb 1st, 2019

[Mark Tinka]

On 31/Jan/19 18:32, James Stahr wrote:

>
> I think the advertised testing tool may be flawed as blocking TCP/53
> is enough to receive a STOP from the dnsflagday web site.  It's been
> my (possibly flawed) understanding that TCP/53 is an option for
> clients but primarily it is a mechanism for the *server* to request
> the client communicate using TCP/53 instead - this could be due to UDP
> response size, anti-spoofing controls, etc...

On a similar note, we tested for all our self-hosted zones OK 2 weeks
ago. However, in previous days, the summary result said "NO GOOD, THINGS
WILL BE SLOW COME FLAG DAY". The detailed test showed IPv4 tested
perfect, but IPv6 probes timed out.

The issue turned out to be an internal IPv6 routing/forwarding issue for
our service within Century Link's (Level(3)'s) network. CL finally fixed
that issue today and the flag day test tool is happy again.

Some of our partners/customers were concerned our name servers were not
ready, based purely on the summary result of the test tool. Perhaps
adding some intelligence about whether the issue is the name server or
the transport may be helpful.

Mark.

Re: Calling LinkedIn, Amazon and Akamai @ DE-CIX NY

[Owen DeLong]

> On Jan 30, 2019, at 17:32 , valdis.kletni...@vt.edu wrote:
> 
> On Wed, 30 Jan 2019 23:55:40 +, "i3D.net - Martijn Schmidt" said:
> 
>> Here: all networks that didn't already change their peering IP are not 
>> yet connected to the updated route-server. Some networks are not 
>> connected to any route-server. Therefore, those networks did not yet 
>> change their peering IP.
>> 
>> I think you can see what's wrong with that statement.. it does not 
>> follow. That has nothing to do with peering department resources, but 
>> everything to do with the chosen peering strategy.
> 
> Under what conditions would somebody be present at the exchange and
> not talking to the route server *at all* before the IP change?

Route servers are a double-edged sword for many networks.

There are a number of reasons that one might choose not to peer with route 
servers at an exchange point, even if you are willing to peer with every single 
individual peer at the exchange.

It would be difficult for me to go into specific details without violating NDAs 
from former employers, but it really doesn’t take all that much imagination.

Consider the following questions:

1.  What information does one get from a direct peering that is 
removed by a route server?
2.  How does the AS PATH change if you are peering with a route 
server?
3.  What tools are available for measuring results of individual 
peering sessions vs. sorting out individual
next-hops learned from a common peering session?

Owen

Re: Latency between Dallas and west coast

[Mike Hammett]

It's 180 ms from Dallas to Djibouti, so no, that much latency to the west coast 
of the US is not normal. 

http://he.net/layer2/ 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Nathanael Catangay Cariaga"  
To: nanog@nanog.org 
Sent: Thursday, January 31, 2019 9:39:54 AM 
Subject: Latency between Dallas and west coast 


I would like to know if anyone here maintains average latency ranges between 
Dallas and Internet Exchanges at the west coast? Is it normal to have around 
192ms to 200ms between the two points? 


Thanks in advance 




-nathan 

Latency between Dallas and west coast

[Nathanael Catangay Cariaga]

I would like to know if anyone here maintains average latency ranges
between Dallas and Internet Exchanges at the west coast?  Is it normal to
have around 192ms  to 200ms between the two points?

Thanks in advance


-nathan

Re: Effects of Cold Front on Internet Infrastructure - U.S. Midwest

[Fletcher Kittredge]

Cold changes the transmission characteristics of fiber. At one point we
were renting some old dark fiber from the local telephone company in
northern Maine. When it would get below -15%-degree F the dB would get bad
enough that the link using that fiber would stop working. The telephone
company was selling us dark fiber because regulation required them to. They
refused to give us another fiber nor inspect/repair. They took the position
they were required to sell us fiber, not working fiber.


On Wed, Jan 30, 2019 at 11:41 AM Mark Tinka  wrote:

> For anyone running IP networks in the Midwest, are you having to do
> anything special to keep your networks up?
>
> For the data centres, is this cold front a chance to reduce air
> conditioning costs, or is it actually straining the infrastructure?
>
> I'm curious, from a +27-degree C summer's day here in Johannesburg.
>
> Mark.
>


-- 
Fletcher Kittredge
GWI
207-602-1134
www.gwi.net

Re: DNS Flag Day, Friday, Feb 1st, 2019

[Owen DeLong]

> On Jan 30, 2019, at 17:40 , Jim Popovitch via NANOG  wrote:
> 
> On Wed, 2019-01-30 at 17:22 -0800, Matthew Petach wrote:
>> Any chance this could wait until say the Tuesday 
>> *after* the Superbowl, when we aren't cutting an 
>> entire religion's worth of potential workers out of 
>> the workforce available to fix issues in case it 
>> turns out to be a bigger problem than is expected, 
>> and when we have less chance of annoying the 
>> vast army of football-loving fans of every sort? 
> 
> IIRC, DNS Flag Day was announce way before last years Super Bowl...
> what did the people who aren't ready for DNS Flag Day do in the past
> 364 days that they need a few more days to get ready for?
> 
> -Jim P.

Consider this…

Sometimes you are responsible for fielding the calls and explaining problems 
that occur on systems that aren’t entirely within your control.

A business class ISP, for example, would have a hard time proactively fixing 
all of their customer’s DNS resolvers and clients. Nonetheless, you can be 
assured that their call center will get the calls when the behavior of DNS 
changes in a way that negatively impacts some fraction of those clients.

In my estimation, the most likely impact of this event will be on the 
enterprise, not the ISP or residential communities.

The ISP community is either aware of and/or dealt with it in the normal course 
of business or they have had their head so deep in the sand that I don’t have 
much sympathy for what happens to them.

The residential end user doesn’t run name servers for the most part, so, 
unlikely to be much impact there. The ones that do (such as myself) are likely 
technical enough and likely sufficiently involved in the ISP community to have 
heard about this issue and taken appropriate action.

In my experience, enterprise IT, OTOH, is widely variable in its attentiveness 
to changes on the internet until after they have occurred. Network focused 
enterprises (e.g. Akamai, DropBox, etc.) are unlikely to be impacted. Other 
kinds of enterprises for whom the internet is more of a utility than a core 
function, OTOH, may have less awareness ahead of time (e.g. Chevron, GM, your 
local dive shop, the bodega on the corner, etc.).

The larger enterprises probably have someone paying some attention. I suspect 
most of the casualties in this event will be in the Small to Medium business 
community.

Owen

Re: DNS Flag Day, Friday, Feb 1st, 2019

[James Stahr]

On 2019-01-31 08:15, Mark Andrews wrote:


We actually have a hard time finding zones where all the servers are
broken enough to not work with servers that don’t fallback to plain
DNS on timeout.  We can find zones where some of the servers are like
that, but there is usually one or more servers that do respond
correctly.

Of the datasets I’ve looked at that 1 in 10,000 to 1 in 100,000 zones
will have problems with updated servers.



I think the advertised testing tool may be flawed as blocking TCP/53 is 
enough to receive a STOP from the dnsflagday web site.  It's been my 
(possibly flawed) understanding that TCP/53 is an option for clients but 
primarily it is a mechanism for the *server* to request the client 
communicate using TCP/53 instead - this could be due to UDP response 
size, anti-spoofing controls, etc...


So is the tool right in saying that TCP/53 is a absolute requirement of 
ENDS0 support for "DNS Flag Day"?  If so, do we expect a dramatic 
increases in TCP/53 requests over UDP/53 queries?  Or is the tool flawed 
in its' claim that the DNS Flag Day changes *require* TCP/53 in order to 
resolve ones domain?  Based upon my reading, the big concern is a 
edns=timeout result (from the ISC tool) not a edns512tcp=timeout.



While I applaud the effort, the message and delivery could have been 
better. My first read on DNS Flag Day was that this was going to be the 
day new software versions were to be released which deprecated EDNS0 
fallback measures so the adoption would be a gradual process by updating 
the latest versions of several DNS servers, only to find out that many 
resolvers used by end users use will be upgrading on Feb 1rst.  Now, it 
sounds like the rollout schedule is on their timetable and could happen 
over the next several weeks and months.  So really not a Flag "Day" now 
is it? I guess that's also why the date being on a Friday also isn't 
really a concern...


Finally, why has no one stepped up to setup an updated resolver prior to 
Feb 1rst for actual testing? Or are there instructions on how one could 
setup their own resolver setup prior to Feb 1rst?  No offense, but I'm 
not jumping to a brand new train of code just to enforce DNS Flag Day 
but I might choose enforce now under *existing* code bases rather than 
waiting for everyone else using Cloudflare/Google/OpenDNS as resolver 
may see me after Feb 1rst?   If anyone has such instructions, post them 
- or put them on dnsflagday.net for anyone else wanting to adopt/test.  
Just some thoughts...



--
-James

RE: Effects of Cold Front on Internet Infrastructure - U.S. Midwest

[Hiers, David]

Excessive cold killed us once when the air exit vents froze shut.


From: NANOG [mailto:nanog-bounces+david.hiers=cdk@nanog.org] On Behalf Of 
Naslund, Steve
Sent: Wednesday, January 30, 2019 9:43 AM
To: nanog@nanog.org
Subject: RE: Effects of Cold Front on Internet Infrastructure - U.S. Midwest

Ironically you don’t really save a lot of energy when it’s this cold because 
the loops are running at high speed and the humidification coils are working 
overtime to keep the RH up in the room.

People think we can bring in all the outside cold we want but the issue then is 
humidity stability.

Steven Naslund
Chicago IL

--
This message and any attachments are intended only for the use of the addressee 
and may contain information that is privileged and confidential. If the reader 
of the message is not the intended recipient or an authorized representative of 
the intended recipient, you are hereby notified that any dissemination of this 
communication is strictly prohibited. If you have received this communication 
in error, notify the sender immediately by return email and delete the message 
and any attachments from your system.

Re: DNS Flag Day, Friday, Feb 1st, 2019

[Mark Andrews]

> On 1 Feb 2019, at 1:21 am, Stephen Satchell  wrote:
> 
> After reading through the thread, this reminds me of the Y2K flap, that
> turned into a non-event.  My checks of authoritative DNS servers for my
> domains show no issues now.

As did I, but if we didn’t try and give lots of notice people would have 
complained.

That said a lot of servers have been fixed that where not fully compliant and 
firewalls
that unnecessarily blocked well formed EDNS packets with one or more of the 
extension mechanism present have been turned off.  The authoritative DNS server 
space is much cleaner now and we
the data to show it.

Mark
-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742  INTERNET: ma...@isc.org

Re: Calling LinkedIn, Amazon and Akamai @ DE-CIX NY

[Mark Tinka]

On 31/Jan/19 16:22, Mike Hammett wrote:

> A prefix is a prefix. A route is a prefix plus a next-hop. Your next
> hop for your PNI is different than your IX.

And for me, it doesn't matter as long as I am maintaining both my public
link sand PNI's properly.

If my peers are not, I'm happy to take the longer path to reach them
until they are sufficiently incentivized to fix that. At the very least,
there won't be packet loss :-).

Mark.

Re: Calling LinkedIn, Amazon and Akamai @ DE-CIX NY

[Mike Hammett]

A prefix is a prefix. A route is a prefix plus a next-hop. Your next hop for 
your PNI is different than your IX. 

I don't believe I advocated running IX links hot. Financially, as an IX 
operator, I'd prefer that people ran all their bits over an IX and that all 
links were best kept below 10% utilization. ;-) Obviously I know that's not 
good engineering or fiscally responsible on the network's behalf. Just going to 
the extreme to support my point. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Mark Tinka"  
To: "Mike Hammett"  
Cc: nanog@nanog.org 
Sent: Thursday, January 31, 2019 8:14:44 AM 
Subject: Re: Calling LinkedIn, Amazon and Akamai @ DE-CIX NY 




On 31/Jan/19 15:54, Mike Hammett wrote: 




Not all routes are created equal. If you have a PNI and an IX connection of 
equal capacity, obviously the IX connection will fill up first given that there 
is more opportunity there. 


I think you meant to say not all "paths" are equal. Routes are routes. Where 
they lead to is another matter. 

The presence of a PNI does not preclude good governance of an exchange point 
link. If you are going to (willingly or otherwise) ignore the health of your 
public peering links over your private ones (or vice versa), then I wish upon 
you all the hell you'll face that comes with taking that position. 

Our policy is simple - 50% utilized, you upgrade. Doesn't matter what type of 
link it is; WDM Transport, IP, peering (public or private), Metro, core 
backbone, protection paths, e.t.c. Choosing to let your public peering links 
run hot because your "major" peers are taken care of by the private links is 
irresponsible. Do a lot of networks do it; hell yes, and for reasons you'd not 
think are obvious. 




Also, there are more moving parts in an IX (and accompanying route servers), 
thus more to go wrong. 



Agreed, but that's not the crux of this thread (even though it's one of the 
reasons we do not relay solely on RS's). 

Mark. 

Re: DNS Flag Day, Friday, Feb 1st, 2019

[Mark Andrews]

> On 31 Jan 2019, at 10:59 pm, Matthew Petach  wrote:
> 
> 
> 
> On Thu, Jan 31, 2019, 01:27 Radu-Adrian Feurdean 
>  
> 
> On Thu, Jan 31, 2019, at 03:24, Mark Andrews wrote:
> > You do realise that when the day was chosen it was just the date after 
> > which new versions of name servers by the original group of Open Source 
> > DNS developers would not have the work arounds incorporated?
> 
> I think it's pretty safe to say that the "DNS Flag day" is more like a date 
> of "end of support" rather than an "service termination". My guess is that 
> some uncompliant servers will be still running long after that date...
> 
> --
> R-A.F. 
> 
> 
> (resending from correct address)
> 
> Right. 
> 
> The concern is that it's *also* the date when all the major recursive lookup 
> servers are changing their behaviour.
> 
> New software availability date?
> Awesome, go for it.
> 
> Google, Cloudflare, Quad9 all changing their codebase/response behaviour on a 
> Friday before a major sporting and advertising event?
> 
> Not sounding like a really great idea from this side of the table. 
> 
> Are we certain that the changes on the part of the big four recursive DNS 
> operators won't cause downstream issues?
> 
> As someone noted earlier, this mainly affects products from a specific 
> company, Microsoft, and L7 load balancers like A10s.  I'm going to hope legal 
> teams from each of the major recursive providers were consulted ahead of time 
> to vet the effort, and ensure there were no concerns about collusion or 
> anticompetitive practices, right?  
> 
> I'm fine with rolling out software that stops supporting bad behaviour.
> 
> What I find to be concerning is when supposedly competing entities all band 
> together in a pact that largely holds the rest of the world hostage to their 
> arbitrary timeline.
> 
> Perhaps it's time to create a new recursive resolver service that explicitly 
> *is not* part of the cabal...
> 
> Matt
> (hoping and praying this weekend will go smoothly)

So you are worrying about sites running Windows DNS from prior to Windows 
Server 2003 (this is where Microsoft added EDNS support) and sites that have a 
firewall that blocks all EDNS queries.  The large recursive server farms don’t 
do DNS COOKIE so you don’t need to worry about that.

Those Windows servers work most of the time even with a DNS servers that don’t 
fall back to plain DNS on timeout.  And if you have installed a firewall that 
blocks EDNS you have shot yourself in the foot.

We actually have a hard time finding zones where all the servers are broken 
enough to not work with servers that don’t fallback to plain DNS on timeout.  
We can find zones where some of the servers are like that, but there is usually 
one or more servers that do respond correctly.

Of the datasets I’ve looked at that 1 in 10,000 to 1 in 100,000 zones will have 
problems with updated servers.

Mark
-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742  INTERNET: ma...@isc.org

Re: Calling LinkedIn, Amazon and Akamai @ DE-CIX NY

[Mark Tinka]

On 31/Jan/19 15:54, Mike Hammett wrote:

> Not all routes are created equal. If you have a PNI and an IX
> connection of equal capacity, obviously the IX connection will fill up
> first given that there is more opportunity there.

I think you meant to say not all "paths" are equal. Routes are routes.
Where they lead to is another matter.

The presence of a PNI does not preclude good governance of an exchange
point link. If you are going to (willingly or otherwise) ignore the
health of your public peering links over your private ones (or vice
versa), then I wish upon you all the hell you'll face that comes with
taking that position.

Our policy is simple - 50% utilized, you upgrade. Doesn't matter what
type of link it is; WDM Transport, IP, peering (public or private),
Metro, core backbone, protection paths, e.t.c. Choosing to let your
public peering links run hot because your "major" peers are taken care
of by the private links is irresponsible. Do a lot of networks do it;
hell yes, and for reasons you'd not think are obvious.

> Also, there are more moving parts in an IX (and accompanying route
> servers), thus more to go wrong.

Agreed, but that's not the crux of this thread (even though it's one of
the reasons we do not relay solely on RS's).

Mark.

Re: DNS Flag Day, Friday, Feb 1st, 2019

[Stephen Satchell]

After reading through the thread, this reminds me of the Y2K flap, that
turned into a non-event.  My checks of authoritative DNS servers for my
domains show no issues now.

Re: Calling LinkedIn, Amazon and Akamai @ DE-CIX NY

[Mike Hammett]

Not all routes are created equal. If you have a PNI and an IX connection of 
equal capacity, obviously the IX connection will fill up first given that there 
is more opportunity there. Also, there are more moving parts in an IX (and 
accompanying route servers), thus more to go wrong. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Mark Tinka"  
To: "Mike Hammett"  
Cc: nanog@nanog.org 
Sent: Thursday, January 31, 2019 7:09:54 AM 
Subject: Re: Calling LinkedIn, Amazon and Akamai @ DE-CIX NY 




On 31/Jan/19 14:59, Mike Hammett wrote: 




Do people not know how to use local pref and MED to prefer PNI over route 
server? 



We don't particularly care how the routes are learned. Routes are routes. 

Our motivation for or against peering with an RS is granular policy control, 
and the level of trust we can put in the stability of the same over time. 

Mark. 

Re: DNS Flag Day, Friday, Feb 1st, 2019

[Jimmy Hess]

On Thu, Jan 31, 2019 at 6:01 AM Matthew Petach  wrote:

> Google, Cloudflare, Quad9 all changing their codebase/response behaviour on a 
> Friday before a major sporting and advertising event?
> Not sounding like a really great idea from this side of the table.

If your DNS zone is hosted on Google or Cloudflare's servers, then you
should have nothing to worry about,  other than your end users having
a broken firewall in between their DNS resolver and the internet, and then
updating their resolver software

Actually, none of those providers announced detailed plans, at least yet;
and maybe they won't even bother announcing.
they could update their software yesterday if they wanted,  or they could
wait until next week,  and it would still be  "On or Around Feb 1, 2019."
The 'Flag Day' is not a specific moment at which all providers
necessarily push a big red button at the same instant to remove
their workaround for broken DNS servers discarding queries.

> Are we certain that the changes on the part of the big four recursive DNS 
> operators won't cause downstream issues?

Not downstream issues.   They will break resolution of  the
domains which have authoritative DNS servers that
discard or ignore DNS queries which comply with all the
original DNS standards but contain EDNS attributes.

The common cause for this was Authoritative DNS servers placed
behind 3rd party Firewalls that tried to "inspect" DNS traffic and
discard queries and responses with "unknown" properties or sizes
larger than 512  ---  there may also be an esoteric DNS proxy/
balancer implementation with bugs, or some broken authoritative
server implementations running software that was more than 10 years
past End of Support at this point.

If your authoritative DNS service sits behind such a broken device or
on such broken DNS server,
then clients already have troubles resolving your domains,  and some time
after the DNS Flag day,  it will likely stop working altogether.

> As someone noted earlier, this mainly affects products from a specific 
> company, Microsoft, and L7 load balancers like A10s.  I'm going to hope legal 
> teams from each of the major recursive providers were consulted ahead of time 
> to vet the effort, and ensure there were no concerns about collusion or 
> anticompetitive practices, right?

I wouldn't be too concerned.The operators of a recursive DNS service
very likely won't have an agreement giving them a duty to  Microsoft,
A10, etc;
If  you have a software or service that you expect to interoperate with DNS
resolvers,  then its on you to make sure your  implementation of the software
or service complies with the agreed standards regarding its processing
of compliant messages.

-- 
-JH

Re: Calling LinkedIn, Amazon and Akamai @ DE-CIX NY

[Mark Tinka]

On 31/Jan/19 14:59, Mike Hammett wrote:

> Do people not know how to use local pref and MED to prefer PNI over
> route server?

We don't particularly care how the routes are learned. Routes are routes.

Our motivation for or against peering with an RS is granular policy
control, and the level of trust we can put in the stability of the same
over time.

Mark.

Re: Calling LinkedIn, Amazon and Akamai @ DE-CIX NY

[Mike Hammett]

Do people not know how to use local pref and MED to prefer PNI over route 
server? 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Mark Tinka"  
To: nanog@nanog.org 
Sent: Thursday, January 31, 2019 6:20:42 AM 
Subject: Re: Calling LinkedIn, Amazon and Akamai @ DE-CIX NY 



On 31/Jan/19 12:04, Julien Goodwin wrote: 

> Even in exchanges that strongly encourage their use route collectors 
> were much less connected to than route servers, and few exchanges had 
> them in the first place. 

We, for example, connect to RS's more selectively. 

We are more liberal about RC's since they do not have an impact on our 
forwarding paradigm, and it helps the exchange point know what's 
happening across their fabric. But yes, I do imagine that interest level 
of connecting to either an RS or RC could vary, particularly the larger 
of a network you are. 

> 
> Part of the problem with advertising on route servers is many clients, 
> including networks that should know better often treat those routes as a 
> higher priority than is sensible, in some cases equal or higher than a 
> PNI link in the same city. 

Well, there are a number of peers that do not have a linear peering 
relationship for all routes available at an exchange point, i.e., they 
don't see those routes both via the RS and bi-lateral sessions. For many 
networks, RS is the true source and bi-lateral sessions are not even 
considered. 

We may not always peer with an RS, but we will always have bi-lateral 
sessions... even when we have sessions to the RS. 

Mark. 

Re: Calling LinkedIn, Amazon and Akamai @ DE-CIX NY

[Mark Tinka]

On 31/Jan/19 12:04, Julien Goodwin wrote:

> Even in exchanges that strongly encourage their use route collectors
> were much less connected to than route servers, and few exchanges had
> them in the first place.

We, for example, connect to RS's more selectively.

We are more liberal about RC's since they do not have an impact on our
forwarding paradigm, and it helps the exchange point know what's
happening across their fabric. But yes, I do imagine that interest level
of connecting to either an RS or RC could vary, particularly the larger
of a network you are.

>
> Part of the problem with advertising on route servers is many clients,
> including networks that should know better often treat those routes as a
> higher priority than is sensible, in some cases equal or higher than a
> PNI link in the same city.

Well, there are a number of peers that do not have a linear peering
relationship for all routes available at an exchange point, i.e., they
don't see those routes both via the RS and bi-lateral sessions. For many
networks, RS is the true source and bi-lateral sessions are not even
considered.

We may not always peer with an RS, but we will always have bi-lateral
sessions... even when we have sessions to the RS.

Mark.

Re: DNS Flag Day, Friday, Feb 1st, 2019

[Matthew Petach]

On Thu, Jan 31, 2019, 01:27 Radu-Adrian Feurdean <
na...@radu-adrian.feurdean.net wrote:

>
>
> On Thu, Jan 31, 2019, at 03:24, Mark Andrews wrote:
> > You do realise that when the day was chosen it was just the date after
> > which new versions of name servers by the original group of Open Source
> > DNS developers would not have the work arounds incorporated?
>
> I think it's pretty safe to say that the "DNS Flag day" is more like a
> date of "end of support" rather than an "service termination". My guess is
> that some uncompliant servers will be still running long after that date...
>
> --
> R-A.F.
>


(resending from correct address)

Right.

The concern is that it's *also* the date when all the major recursive
lookup servers are changing their behaviour.

New software availability date?
Awesome, go for it.

Google, Cloudflare, Quad9 all changing their codebase/response behaviour on
a Friday before a major sporting and advertising event?

Not sounding like a really great idea from this side of the table.

Are we certain that the changes on the part of the big four recursive DNS
operators won't cause downstream issues?

As someone noted earlier, this mainly affects products from a specific
company, Microsoft, and L7 load balancers like A10s.  I'm going to hope
legal teams from each of the major recursive providers were consulted ahead
of time to vet the effort, and ensure there were no concerns about
collusion or anticompetitive practices, right?

I'm fine with rolling out software that stops supporting bad behaviour.

What I find to be concerning is when supposedly competing entities all band
together in a pact that largely holds the rest of the world hostage to
their arbitrary timeline.

Perhaps it's time to create a new recursive resolver service that
explicitly *is not* part of the cabal...

Matt
(hoping and praying this weekend will go smoothly)

Re: DNS Flag Day, Friday, Feb 1st, 2019

[Mark Andrews]

-- 
Mark Andrews

> On 31 Jan 2019, at 20:25, Radu-Adrian Feurdean 
>  wrote:
> 
> 
> 
>> On Thu, Jan 31, 2019, at 03:24, Mark Andrews wrote:
>> You do realise that when the day was chosen it was just the date after 
>> which new versions of name servers by the original group of Open Source 
>> DNS developers would not have the work arounds incorporated?
> 
> I think it's pretty safe to say that the "DNS Flag day" is more like a date 
> of "end of support" rather than an "service termination". My guess is that 
> some uncompliant servers will be still running long after that date...
> 
> --
> R-A.F. 

Re: Calling LinkedIn, Amazon and Akamai @ DE-CIX NY

[Julien Goodwin]

On 31/1/19 7:08 pm, Mark Tinka wrote:
> I believe most exchange points maintain both route servers and route
> collectors.
> 
> Generally, most peers will connect to the RS, but not all. As you
> mention, some may connect but not send any routes.
> 
> However, I believe all peers will connect to the RC. Of course, I can
> envisage scenarios where even this could be selectively done. But the
> reasons for (not) connecting to the RC are very different from those for
> (not) connecting to the RS.

Back when I looked more deeply at this (mid-2014 when I was redoing the
AS15169 policy around this area) I saw things rather differently, and my
impression is little has changed since.

Even in exchanges that strongly encourage their use route collectors
were much less connected to than route servers, and few exchanges had
them in the first place.

Part of the problem with advertising on route servers is many clients,
including networks that should know better often treat those routes as a
higher priority than is sensible, in some cases equal or higher than a
PNI link in the same city.

Yes we could have used prepends, but that's not necessarily effective
and affects everyone for the actions of a few, and is why the routes
AS15169 advertised on route servers reduced back then.

Re: DNS Flag Day, Friday, Feb 1st, 2019

[Radu-Adrian Feurdean]

On Thu, Jan 31, 2019, at 03:24, Mark Andrews wrote:
> You do realise that when the day was chosen it was just the date after 
> which new versions of name servers by the original group of Open Source 
> DNS developers would not have the work arounds incorporated?

I think it's pretty safe to say that the "DNS Flag day" is more like a date of 
"end of support" rather than an "service termination". My guess is that some 
uncompliant servers will be still running long after that date...

--
R-A.F. 

Re: BGP Experiment

[Saku Ytti]

Hey,

> This is because you did your due diligence during the testing.
> Do you have statistics on the probability of these "complex" bugs occurrence?

No. I wish I had and I hope to make change on this. Try to translate
how good investment test is, how many customer outages it has saved
etc.

I suspect simple bugs are found by vendor, complex bugs are not
economic to find. And testing is more proof of work than business
case.

-- 
  ++ytti

RE: BGP Experiment

[adamv0025]

> From: Saku Ytti 
> Sent: Friday, January 25, 2019 7:59 AM
> 
> On Thu, 24 Jan 2019 at 18:43,  wrote:
> 
> > We fight with that all the time,
> > I'd say that from the whole Design->Certify->Deploy->Verify->Monitor
> service lifecycle time budget, the service certification testing is almost 
> half of
> it.
> > That's why I'm so interested in a model driven design and testing approach.
> 
> This shop has 100% automated blackbox testing, and still they have to cherry-
> pick what to test. 
>
Sure one tests only for the few specific current and near future use cases.

> Do you have statistics how often you find show-stopper
> issues and how far into the test they were found? 
>
I don't keep those statistics, but running bug scrubs in order to determine the 
code for regression testing is usually good starting point to avoid 
show-stoppers, what is then found later on during the testing is usually 
patched -so yes you end up with a brand new code and several patches related to 
your use cases (PEs, Ps, etc..)
   
> I expect this to be
> exponential curve, like upgrading box, getting your signalling protocols up,
> pushing one packet in each service you sell is easy and fast, I wonder will
> massive amount of work increase confidence significantly from that. 
>
Yes it will.

> The
> issues I tend to find in production are issues which are not trivial to 
> recreate
> in lab, once we know what they are, which implies that finding them a-priori
> is bit naive expectation. So, assumptions:
>
This is because you did your due diligence during the testing. 
Do you have statistics on the probability of these "complex" bugs occurrence?

> Hopefully we'll enter NOS future where we download NOS from github and
> compile it to our devices. Allowing whole community to contribute to unit
> testing and use-cases and to run minimal bug surface code in your
> environment.
>
Not there yet, but you can compile your own routing protocols and run those on 
vendor OS.

> I see very little future in blackbox testing vendor NOS at operator site,
> beyond quick poke at lab. Seems like poor value. Rather have pessimistic
> deployment plan, lab => staging => 2-3 low risk site =>
> 2-3 high risk site => slow roll up
> 
Yes that's also a possibility -one of the strong arguments for massive 
disaggregation at the edge, to reduce the fallout of a potential critical 
failure.
Depends on the shop really.

> > I really need to have this ever growing library of test cases that the 
> > automat
> will churn through with very little human intervention, in order to reduce the
> testing from months to days or weeks at least.
> 
> Lot of vendor, maybe all, accept your configuration and test them for
> releases. I think this is only viable solution vendors have for blackbox, 
> gather
> configs from customers and test those, instead of try to guess what to test.
> I've done that with Cisco in two companies, unfortunately I can't really tell 
> if it
> impacted quality, but I like to think it did.
> 
Did that with juniper partners and now directly with Juniper. 
The thing is though they are using our test plan...

adam  

Re: Calling LinkedIn, Amazon and Akamai @ DE-CIX NY

[Mark Tinka]

On 31/Jan/19 09:51, Thomas King wrote:

> Hi Ren et al.,
>
> Thanks for pointing out that some peers do not use the route servers. This 
> group can be subdivided in a group of peers not sending any IP prefixes to 
> the route servers while maintaining a route server BGP session, and a group 
> of peers not even connecting to the route server. The latter do not show up 
> in the "BGP session established" section even if they have applied the 
> required IPv4 changes.

I believe most exchange points maintain both route servers and route
collectors.

Generally, most peers will connect to the RS, but not all. As you
mention, some may connect but not send any routes.

However, I believe all peers will connect to the RC. Of course, I can
envisage scenarios where even this could be selectively done. But the
reasons for (not) connecting to the RC are very different from those for
(not) connecting to the RS.

Mark.