Re: Google DNS intermittent ServFail for Disney subdomain

[David Sotnick]

Well well, it looks like a Direct Connect circuit to Google was leaking the
route to this DMZ 153.7.233.0/24 back to Google via BGP.

Return traffic from Google (for only some fraction of DNS queries) was
passing back across this leaked route, and being dropped on this Direct
Connect peering point at Disney.

Gotta love it when a problem is solved, by the OP, within an hour of
resorting to mailing the NANOG community.

Thanks all, nothing to see here!

-David

On Thu, Oct 19, 2017 at 8:41 PM, David Sotnick <sotnickd-na...@ddv.com>
wrote:

> Hi Nanog,
>
> I am principal network engineer for sister-studio to Disney Studios. They
> have been struggling with DNS issues since Thursday 12th October.
>
> By all accounts it appears as though *some* of the Google DNS resolvers
> cannot reach the authoritative nameservers for "studio.disney.com".
>
> This is causing ~20-30% of all DNS requests against Google Public DNS
> 8.8.8.8 / 8.8.4.4 to fail for requests in this subdomain.
>
> The name servers reside in 153.7.233.0/24.
>
> Might someone be able to *connect me* with someone at Google to assist my
> poor colleagues who are banging their heads against a brick wall here.
>
> Thank you,
> David
>

Google DNS intermittent ServFail for Disney subdomain

[David Sotnick]

Hi Nanog,

I am principal network engineer for sister-studio to Disney Studios. They
have been struggling with DNS issues since Thursday 12th October.

By all accounts it appears as though *some* of the Google DNS resolvers
cannot reach the authoritative nameservers for "studio.disney.com".

This is causing ~20-30% of all DNS requests against Google Public DNS
8.8.8.8 / 8.8.4.4 to fail for requests in this subdomain.

The name servers reside in 153.7.233.0/24.

Might someone be able to *connect me* with someone at Google to assist my
poor colleagues who are banging their heads against a brick wall here.

Thank you,
David

Re: Updating Geolocation of /24 within corporate /16

[David Sotnick]

Hi Tyler,

I have not yet tried this, but am doing so now, thanks!

-Dave

On Thu, Feb 9, 2017 at 6:27 PM, Tyler Conrad <ty...@tgconrad.com> wrote:

> Have you tried submitting a correction to some geolocation services
> directly yet? Maxmind is pretty heavily used.
>
> https://support.maxmind.com/correction-faq/submit-a-
> correction/how-do-i-submit-a-correction-to-geoip-data/
>
>
> On Thursday, February 9, 2017, David Sotnick <sotnickd-na...@ddv.com>
> wrote:
>
>> Hi NANOG,
>>
>> You have given good advice on updating IP Geolocation data in the past,
>> including visiting 'www.google.com' from a mobile device and selecting
>> "use
>> exact location [from GPS]". This worked out well for us a few years ago
>> for
>> a single IP which we were NATting out of in a new geographic location.
>>
>> Now we are in a position where we have been assigned site-local /24 (out
>> of
>> the corporation's larger /20 space) networks for a couple of locations and
>> I'm wondering how I go about updating IP Geolocation data to note that two
>> /24 networks are no longer at the Corporate HQ location.
>>
>> I understand that when users first start using these site-specific /24
>> networks, they will be lumped in with the larger /20 space as far as their
>> geolocation goes, but besides the Google/GPS method, is there a
>> cleaner/better way to do this? Do Geolocation services use SWIP data?
>> Should I have the /24s have separate SWIP data noting the geo location?
>> I'd
>> love a place to be able to say: "This /24 is at this geoloc; this /24 is
>> at
>> this geoloc; and the corporate /20 remains where it always has been."
>>
>> Many thanks for your insights in this matter,
>>
>> -Dave
>>
>

Updating Geolocation of /24 within corporate /16

[David Sotnick]

Hi NANOG,

You have given good advice on updating IP Geolocation data in the past,
including visiting 'www.google.com' from a mobile device and selecting "use
exact location [from GPS]". This worked out well for us a few years ago for
a single IP which we were NATting out of in a new geographic location.

Now we are in a position where we have been assigned site-local /24 (out of
the corporation's larger /20 space) networks for a couple of locations and
I'm wondering how I go about updating IP Geolocation data to note that two
/24 networks are no longer at the Corporate HQ location.

I understand that when users first start using these site-specific /24
networks, they will be lumped in with the larger /20 space as far as their
geolocation goes, but besides the Google/GPS method, is there a
cleaner/better way to do this? Do Geolocation services use SWIP data?
Should I have the /24s have separate SWIP data noting the geo location? I'd
love a place to be able to say: "This /24 is at this geoloc; this /24 is at
this geoloc; and the corporate /20 remains where it always has been."

Many thanks for your insights in this matter,

-Dave

Re: IPv6-enabled multi-factor providers (not DUO)

[David Sotnick]

Ahhh, a recent development — that's great news! Yes, it should be
exceedingly simple for Duo to implement IPv6 :-)

Thanks for the edification, all!

-Dave

On Thu, Feb 2, 2017 at 11:41 AM, Christopher Morrow <morrowc.li...@gmail.com
> wrote:

>
>
> On Thu, Feb 2, 2017 at 11:32 AM, David Sotnick <sotnickd-na...@ddv.com>
> wrote:
>
>> Hi NANOG,
>>
>> (Apologies if this is slightly off-topic; there are a lot of
>> IPv6-advocates
>> here who might have some insights).
>>
>> At my day job, we use Duo Security for MFA. It works well, with the
>> caveats
>> that it's cloud-based and heavily dependent on Amazon AWS.
>>
>>
> https://aws.amazon.com/blogs/aws/new-ipv6-support-for-ec2-
> instances-in-virtual-private-clouds/
>
> should be all set, right?
>
>
>> We had an IPv4 outage last weekend and of course our Duo MFA only supports
>> IPv4 and given their dependence on AWS I'm not hopeful that they'll have
>> IPv6 access to their API servers any time this year. If they had, my
>> weekend would have gone a lot easier.
>>
>> What MFA alternatives are out there that support IPv6? If we found a
>> suitable alternative, I'm sure we'd consider shifting our business their
>> way.
>>
>> Thanks!
>>
>> -Dave
>>
>
>

IPv6-enabled multi-factor providers (not DUO)

[David Sotnick]

Hi NANOG,

(Apologies if this is slightly off-topic; there are a lot of IPv6-advocates
here who might have some insights).

At my day job, we use Duo Security for MFA. It works well, with the caveats
that it's cloud-based and heavily dependent on Amazon AWS.

We had an IPv4 outage last weekend and of course our Duo MFA only supports
IPv4 and given their dependence on AWS I'm not hopeful that they'll have
IPv6 access to their API servers any time this year. If they had, my
weekend would have gone a lot easier.

What MFA alternatives are out there that support IPv6? If we found a
suitable alternative, I'm sure we'd consider shifting our business their
way.

Thanks!

-Dave

Re: Need Comcast IPv6 routing assistance please

[David Sotnick]

Hi John,

I have been working with Courtney Smith and a fix has been implemented.
Apparently a bunch of new Level(3) peering circuits were turned up on 5/15
and that's when the chronic packet loss problem started for our users.

I have not been informed of the details as to what was causing such packet
loss (but I would love to know), but for now the problem is resolved.

FWIW, this problem doesn't appear limited to the Northern CA region, as we
have users in Seattle, WA (who VPN down to Northern CA), and their packet
loss issues have also been resolved.

I don't see two delegated prefixes and besides wouldn't that particular
issue need to be present on all our users' Comcast connections in order for
them *all* to have experienced the same packet loss? I think perhaps that's
a red-herring.

Cheers,
David

On Tue, May 24, 2016 at 3:23 AM, Brzozowski, John <
john_brzozow...@cable.comcast.com> wrote:

> Regarding the thread:
>
> http://mailman.nanog.org/pipermail/nanog/2016-May/085878.html
>
> David,
>
> I looked around CA and it looks like some customers are provisioned with
> two delegated IPv6 prefixes.  We had an issue a week or so back that we
> believe was corrected.  If you wish contact me off list.
>
> Before we look to see if there are larger routing issue we should make
> sure you have one and only one active delegated IPv6 prefix.  From my end
> it looks like you may have two.
>
> Thanks,
>
> John
> +1-484-962-0060
>
>
>

Need Comcast IPv6 routing assistance please

[David Sotnick]

Hello NANOG,

Could someone from Comcast IPv6 routing team please contact me directly? I
am both a business and residential comcast customer and my employer is a
Level(3) HSIP customer at multiple sites.

I'm seeing *consistent* 46.1% packet loss between Comcast Res/Bus services
in Northern CA and Pixar (Level 3 customer) also in Northern CA. I have
ticket open with Level (3) but the problem appears to be on Comcast's
network.

Sample trace:

   My traceroute  [v0.85]
ipv6testhost.ddv.com (::)
 Mon May 23 10:56:05 2016
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
   Packets
  Pings
 HostLoss%   Snt   Last
  Avg  Best  Wrst StDev
 1. 2601:647:280:23::10.0%   4640.6
  0.4   0.3   8.7   0.4
 2. 2001:558:4000:3d::1   0.2%   463   13.0
 10.3   8.2  27.4   2.1
 3. te-0-7-0-5-sur03.sanrafael.ca.sfba.comcast.net   10.2%   4639.6
 10.7   8.5  34.8   2.4
 4. be-207-rar01.rohnertpr.ca.sfba.comcast.net   44.7%   463   10.6
 11.7   9.4  25.9   2.1
 5. he-0-18-0-0-ar01.santaclara.ca.sfba.comcast.net  51.6%   463   15.2
 14.1  12.0  26.2   1.9
 6. 2001:1900:4:3::439   46.0%   463   13.3
 14.4  11.8  50.2   3.6
 7. vl-80.edge1.SanJose1.Level3.net  44.9%   463   12.1
 13.9  11.7  28.5   2.3
 8. vl-4045.edge5.LosAngeles.Level3.net  45.4%   463   21.2
 21.5  19.2  39.4   2.6
 9. vl-4044.bar1.LasVegas1.Level3.net46.4%   463   24.9
 27.7  24.4  88.3   6.8
10. vl-5.car1.LasVegas1.Level3.net   46.2%   463  104.3
 46.3  24.5 318.2  48.0
11. PIXAR-ANIMA.car1.LasVegas1.Level3.net44.9%   463   27.6
 27.4  25.0  37.7   2.1
12. 2620:79:0:b04d::249  45.1%   463   46.4
 48.9  46.0 114.2   4.9

And pings back from Pixar:

Type escape sequence to abort.Sending 500, 100-byte ICMP Echos to
2601:647:0:1900:242:DEA1:FEC9:FFAE, timeout is 2 seconds:
Packet sent with a source address of
2620:79:0:B04D::249%internet.!
!..!...!...!!!..!!!...!!
!!!..!.!!!..!.!!.....!!!
!..!!...!...!..!!..!!...
!!Success rate is 90 percent (452/500), round-trip min/avg/max =
12/30/68 ms

Any help really appreciated as you can imagine how painful remote access
for our employees with Comcast connections into Pixar over IPv6 is right
now.

Many Thanks,
David

Re: 10G-capable customer router recommendations?

[David Sotnick]

Thanks for the replies, everyone! Much appreciated.

I'm going to check out the Mikrotik CCR series out.

Cheers,
Dave

On Fri, Apr 15, 2016 at 1:18 PM, David Sotnick <sotnickd-na...@ddv.com>
wrote:

> Hello masters of the Internet,
>
> I was recently asked to set up networking at a VIP's home where he has
> Comcast "Gigabit Pro" service, which is delivered on a 10G-SR MM port on a
> Comcast-supplied Juniper ACX-2100 router.
>
> Which customer router would you suggest for such a setup? It needs to do
> IPv4 NAT, DHCP, IPv4+IPv6 routing and have a decent L4 firewall (that also
> supports IPv6).
>
> The customer pays for "2Gb" service (Comcast caps this at 2G+10% =
> 2.2Gbps) and would like to get what he pays for (*cough*) by having the
> ability to stream two 1Gbps streams (or at least achieve > 1.0Gbps).
>
> I'm tempted to get another ACX-2100 and do a 4x1Gb LACP port-channel to
> the customer switch, or replace the AV-integrator-installed Cisco SG300-52P
> (Cisco switch with e.g. an EX-3300 with 10Gb uplinks).
>
> Thanks in advance for your suggestions.
>
> -Dave
>

Re: 10G-capable customer router recommendations?

[David Sotnick]

Thanks Aaron. Unless something has changed recently, I don't think the
Brocade ICX series does NAT either.

On Fri, Apr 15, 2016 at 2:52 PM, Aaron  wrote:

> Not a lot of 10G capable CPEs out there.  For our 10G residential
> customers we install Brocade ICXs.
>
> Aaron
>
> --
> 
> Aaron Wendel
> Chief Technical Officer
> Wholesale Internet, Inc. (AS 32097)
> (816)550-9030
> http://www.wholesaleinternet.com
> 
>
>

10G-capable customer router recommendations?

[David Sotnick]

Hello masters of the Internet,

I was recently asked to set up networking at a VIP's home where he has
Comcast "Gigabit Pro" service, which is delivered on a 10G-SR MM port on a
Comcast-supplied Juniper ACX-2100 router.

Which customer router would you suggest for such a setup? It needs to do
IPv4 NAT, DHCP, IPv4+IPv6 routing and have a decent L4 firewall (that also
supports IPv6).

The customer pays for "2Gb" service (Comcast caps this at 2G+10% = 2.2Gbps)
and would like to get what he pays for (*cough*) by having the ability to
stream two 1Gbps streams (or at least achieve > 1.0Gbps).

I'm tempted to get another ACX-2100 and do a 4x1Gb LACP port-channel to the
customer switch, or replace the AV-integrator-installed Cisco SG300-52P
(Cisco switch with e.g. an EX-3300 with 10Gb uplinks).

Thanks in advance for your suggestions.

-Dave

Re: How to update IPv6 geolocation data? Google sites blocked.

[David Sotnick]

Hi Hugo,

Thanks for the follow-up. For some reason both responses from Mr. Lewis
ended up my Gmail (domain) Spam folder. I have never had a NANOG response
go into Spam, so I didn't even think to check there.

I'll give this a shot today. Thanks again!

-David


On Wed, Jan 6, 2016 at 9:18 PM, Hugo Slabbert <h...@slabnet.com> wrote:

> On Wed 2016-Jan-06 16:23:21 -0800, David Sotnick <sotnickd-na...@ddv.com>
> wrote:
>
> Really? Nobody here knows how one goes about updating IPv6 geolocation
>> data? Our /48 is still being denied access to Google sites due to unknown
>> geolocation.
>>
>> Help?
>>
>
> John Lewis responded with some info[1], which is backed up by Google's own
> support page[2][3].  No bets from me on how quickly or reliably that gets
> updated, though...
>
>
>> Best,
>> David
>>
>
> --
> Hugo
>
> h...@slabnet.com: email, xmpp/jabber
> PGP fingerprint (B178313E):
> CF18 15FA 9FE4 0CD1 2319 1D77 9AB1 0FFD B178 313E
>
> (also on Signal)
>
> [1] http://mailman.nanog.org/pipermail/nanog/2015-December/083078.html
> [2] https://support.google.com/websearch/answer/873?hl=en
> [3] https://support.google.com/websearch/answer/179386
>
>
> On Tue, Dec 22, 2015 at 1:34 PM, David Sotnick <sotnickd-na...@ddv.com>
>> wrote:
>>
>> Hello, and Season's Greetings!
>>>
>>> We recently lit up a new IPv6-connected location and expanded our
>>> ARIN-allocated /48 network to a /44 network to accommodate the additional
>>> location (and future locations).
>>>
>>> However, since moving our small satellite office off our primary /48 and
>>> onto their own /48 as part of our /44 network, the users at that office
>>> are
>>> receiving messages from e.g. YouTube that the "user has not made this
>>> content available in your country".
>>>
>>> How does one go about updating this v6 geolocation data? This is
>>> impacting
>>> a bunch of our users.
>>>
>>> Thanks!
>>>
>>> -David
>>>
>>>

Re: How to update IPv6 geolocation data? Google sites blocked.

[David Sotnick]

Really? Nobody here knows how one goes about updating IPv6 geolocation
data? Our /48 is still being denied access to Google sites due to unknown
geolocation.

Help?

Best,
David

On Tue, Dec 22, 2015 at 1:34 PM, David Sotnick <sotnickd-na...@ddv.com>
wrote:

> Hello, and Season's Greetings!
>
> We recently lit up a new IPv6-connected location and expanded our
> ARIN-allocated /48 network to a /44 network to accommodate the additional
> location (and future locations).
>
> However, since moving our small satellite office off our primary /48 and
> onto their own /48 as part of our /44 network, the users at that office are
> receiving messages from e.g. YouTube that the "user has not made this
> content available in your country".
>
> How does one go about updating this v6 geolocation data? This is impacting
> a bunch of our users.
>
> Thanks!
>
> -David
>

How to update IPv6 geolocation data? Google sites blocked.

[David Sotnick]

Hello, and Season's Greetings!

We recently lit up a new IPv6-connected location and expanded our
ARIN-allocated /48 network to a /44 network to accommodate the additional
location (and future locations).

However, since moving our small satellite office off our primary /48 and
onto their own /48 as part of our /44 network, the users at that office are
receiving messages from e.g. YouTube that the "user has not made this
content available in your country".

How does one go about updating this v6 geolocation data? This is impacting
a bunch of our users.

Thanks!

-David

Joker.com contact / GLUE help

[David Sotnick]

Hi NANOG,

Does anyone have any technical contacts at Joker.com? I am going in circles
with their support folks trying to update the GLUE records for two of my
nameservers and keep running into permissions issues despite the glue
records clearly being part of my domain.

I need to speak to someone who actually understands what DNS Glue records
are, and how to go about updating them.

I feel like the joke is on me for choosing Joker.com as their support has
been laughable thus far.

TIA,
David

Multiple vendors' IPv6 issues

[David Sotnick]

Hi NANOG,

The company I work for has no business case for being on the IPv6-Internet.
However, I am an inquisitive person and I am always looking to learn new
things, so about 3 years ago I started down the IPv6 path. This was early
2012.

Fast forward to today. We have a /44 presence for our company's multiple
sites; All our desktop computers have been on the IPv6 Internet since June,
2012 and we have a few s in our external DNS for some key services —
and, there have been bugs. *Lots* of bugs.

Now, maybe (_maybe_) I can have some sympathy for smaller network companies
(like Arista Networks at the time) to not quite have their act together as
far as IPv6 goes, but for larger, well-established companies to still have
critical IPv6 bugs is just inexcusable!

This month has just been the most disheartening time working with IPv6.

Vendor 1:

Aruba Networks. Upon adding an IPv6 address to start managing our WiFi
controller over IPv6, I receive a call from our Telecom Lead saying that or
WiFi VoIP phones have just gone offline. WHAT? All I did was add an IPv6
address to a management interface which has *nothing* to do with our VoIP
system or SSID, ACLs, policies, roles, etc.

Vendor 2:

Palo Alto Networks: After upgrading our firewalls from a version which has
a nasty bug where the IPv6 neighbor table wasn't being cleaned up properly
(which would overflow the table and break IPv6), we now have a *new* IPv6
neighbor discovery bug where one of our V6-enabled DMZ hosts just falls of
the IPv6 network. The only solution: clear the neighbor table on the Palo
Alto or the client (linux) host.

Vendor 3:

Arista Networks: We are seeing a very similar ND bug with Arista. This one
is slightly more interesting because it only started after upgrading our
Arista EOS code — and it only appears to affect Virtual Machines which are
behind our RedHat Enterprise Virtualization cluster. None of the hundreds
of VMware-connected hosts are affected. The symptom is basically the same
as the Palo Alto bug. Neighbor table gets in some weird state where ND
breaks and the host is unreachable until the neighbor table is cleared.

Oh, and the final straw today, which is *almost* leading me to throw in the
IPv6 towel completely (for now): On certain hosts (VMs), scp'ing a file
over the [Arista] LAN (10 gigabit LAN) takes 5 minutes over IPv6 and 1
second over IPv4. What happened?

It really saddens me that it is still not receiving anywhere near the kind
of QA (partly as a result of lack of adoption) that IPv4 has.

Oh, and let's not forget everybody's favorite vendor, Cisco. Why is it,
Cisco, that I have to restart my IPv6 OSPF3 process on my ASA every time my
Palo Alto firewall crashes and fails over, otherwise none of my VPN clients
can connect via IPv6?

Why do you hurt me so, IPv6? I just wanted to be friends, and now I just
want to break up with you. Maybe we can try to be friends again when your
vendors get their shit together.

-David

Sub-optimal routing to Google via IPv6

[David Sotnick]

I have noticed that since we deployed IPv6 a number of years ago, that our
IPv6 routes to Google's V6-enabled sites (e.g. www.google.com and
www.youtube.com) traverse the CONUS from Oakland (where our primary Level 3
ISP connection is) to Washington D.C., New York, and then onto Google's
network in New York, where the packets presumably pass across Google's
internal networks.

   My traceroute  [v0.71]

hivemind (::)
Thu Feb 26 18:03:44 2015

Keys:  Help   Display mode   Restart statistics   Order of fields   quit

  Packets
Pings

 Host   Loss%  Last
  Avg  Best  Wrst StDev

 1. 2620:79:0:::ff7d 0.0%   0.4
  0.4   0.4   0.4   0.0

 2. 2620:79:0:::fd   0.0%   0.4
  0.4   0.4   0.4   0.0

 3. 2620:79:0:::249  0.0%   1.7
  1.7   1.7   1.7   0.0

 4. ge-6-24.car1.Oakland1.Level3.net 0.0% 316.3
316.3 316.3 316.3   0.0

 5. vl-4043.edge1.SanJose1.Level3.net0.0%   3.0
  3.0   3.0   3.0   0.0

 6. vl-4045.edge5.LosAngeles.Level3.net  0.0%   9.3
  9.3   9.3   9.3   0.0

 7. vl-4081.edge6.LosAngeles1.Level3.net 0.0%   9.2
  9.2   9.2   9.2   0.0

 8. vl-4041.edge1.Washington1.Level3.net 0.0% 116.5
116.5 116.5 116.5   0.0

 9. vl-4080.edge2.Washington1.Level3.net 0.0%
75.0  75.0  75.0  75.0   0.0

10. vl-4068.edge2.Washington12.Level3.net0.0%
75.5  75.5  75.5  75.5   0.0

11. vl-4047.car1.NewYork1.Level3.net 0.0%
76.5  76.5  76.5  76.5   0.0

12. vl-60.ear2.NewYork1.Level3.net   0.0% 110.2
110.2 110.2 110.2   0.0

13. Google-level3-30GB.NewYork1.Level3.net   0.0%
75.6  75.6  75.6  75.6   0.0

14. 2001:4860::1:0:3be   0.0%
76.1  76.1  76.1  76.1   0.0

15. 2001:4860::8:0:4397  0.0%
75.9  75.9  75.9  75.9   0.0

16. 2001:4860::8:0:5901  0.0%
73.5  73.5  73.5  73.5   0.0

17. 2001:4860::8:0:7894  0.0%
85.9  85.9  85.9  85.9   0.0

18. 2001:4860::8:0:79e5  0.0%
92.9  92.9  92.9  92.9   0.0

19. 2001:4860::8:0:6117  0.0%
73.5  73.5  73.5  73.5   0.0

20. 2001:4860::1:0:7ea   0.0%
71.9  71.9  71.9  71.9   0.0

21. 2001:4860:0:1::691   0.0%
72.1  72.1  72.1  72.1   0.0

22. ???

I haven't raised this issue with Level(3) yet, as I was wondering if this
is really a Level(3) routing issue or a Google IPv6 routing issue?

Thank for any insights.

Regards,
David Sotnick
--
Pixar
Emeryville, CA

Comcast Business IPv6 issues

[David Sotnick]

Hi NANOG,

I am a Comcast business Internet subscriber and have been struggling with
having my assigned IPv6 /64 block changing every time Comcast pushes out a
firmware update to my (Motorola BitSurfer) CM.

It seems rather silly that my IPv4 address has not changed in the six
months I've been a Comcast business customer, yet my IPv6 address changes
every few weeks, always after a reset command is sent from the head-end to
my CM (or after power outage).

My *home* Comcast IPv6 address has not changed in over a year, and the same
applies for my IPv4 address, so something is different about the way
Comcast is treating their IPv6 business customers.

This is getting quite frustrating as I run my security cameras over IPv6
and when the addresses change out from under me, things obviously break.

Any insights here from the wise hivemind of Nanog or from Comcast network
folks? Any chance static IPv6 will be available to business customers?

While I am on the subject — ip6.arpa reverse DNS delegation would also be a
nice thing to have if and when static IPv6 addressing is made available. :)

Thanks!

David Sotnick
--
DDV Studios

Severe latency at both San Jose and Los Angeles Level3/ATT peering

[David Sotnick]

Hi Nanog,

I have a ticket open with Level 3, with whom I have 1gig pipes in Oakland,
CA and Las Vegas, NV.

One of our users noticed very slow file transfer/media delivery from the
Bay Area to L.A., and on investigating it appears as though the peering
point between Level3 and ATT in SF was saturated and had 300ms avg.
latency.

90 minutes later after receiving no call from Level3, I escalated to a P1
ticket, as the latency is now  1000ms and we're seeing 20% packet loss.

I decided to statically route to the destination via our DR cluster in
Vegas, and interestingly I found the same situation where ATT and Level3
peer in Tustin.

mtr traceroutes, for those curious:

Via Oakland:

   My traceroute  [v0.71]

hivemind (0.0.0.0)
Fri Apr 11 15:36:08 2014

Keys:  Help   Display mode   Restart statistics   Order of fields   quit

  Packets
Pings

 Host   Loss%  Last
  Avg  Best  Wrst StDev

 1. 138.72.xxx.xxx   0.0%   0.3
  0.2   0.1   0.3   0.1
 2. pan5060-ae1-401.routerland.pixar.com 0.0%   0.4
  0.4   0.4   0.5   0.0
 3. verge-vlan66.pixar.com   0.0%   0.6
  0.7   0.6   0.9   0.1
 4. ge-6-24.car1.Oakland1.Level3.net 0.0%   0.7
105.5   0.7 307.3 110.9
 5. ae-5-5.ebr2.SanJose1.Level3.net  0.0%   1.7
  1.7   1.6   2.8   0.4
 6. ae-92-92.csw4.SanJose1.Level3.net0.0%   1.6
  1.7   1.6   3.0   0.4
 7. ae-4-90.edge2.SanJose1.Level3.net0.0%   1.6
  4.6   1.6  37.1  10.2
 8. 192.205.32.209  41.7% 1042.
1048. 1038. 1059.   9.1
 9. cr1.sffca.ip.att.net25.0% 1052.
1059. 1046. 1072.  10.0
10. cr1.la2ca.ip.att.net27.3% 1043.
1060. 1043. 1071.  10.7
11. cr83.la2ca.ip.att.net   16.7% 1058.
1060. 1045. 1073.   8.8
12. gar7.la2ca.ip.att.net   16.7% 1059.
1061. 1044. 1087.  13.3
13. 12.249.143.98   33.3% 1059.
1057. 1048. 1071.   7.8
14. ???

   My traceroute  [v0.71]

hivemind (0.0.0.0)
Fri Apr 11 15:36:43 2014

Resolver: Received error response 2. (server failure)er of fields   quit

  Packets
Pings

 Host   Loss%  Last
  Avg  Best  Wrst StDev
 1. 138.72.xxx.xxx   0.0%   0.2
  0.1   0.1   0.2   0.0
 2. pan5060-ae1-401.routerland.pixar.com 0.0%   0.4
  0.4   0.3   0.6   0.1
 3. cat-vegas-01-vlan66.pixar.com0.0%  22.0
 21.8  21.7  22.3   0.2
 4. 205.129.21.101   0.0%  19.4
 19.5  19.3  19.9   0.2
 5. ae-2-5.bar1.LasVegas1.Level3.net 0.0%  19.3
 21.8  19.3  40.7   5.9
 6. ae-4-4.ebr1.LosAngeles1.Level3.net   0.0%  22.0
 22.4  21.9  26.8   1.3
 7. ae-6-6.ebr1.Tustin1.Level3.net   0.0%  20.0
 20.2  19.9  21.8   0.5
 8. ae-107-3507.bar2.Tustin1.Level3.net  0.0%  22.0
 22.0  21.9  22.1   0.0
 9. 192.205.37.145  30.8% 1052.
1063. 1048. 1072.   8.1
10. cr1.la2ca.ip.att.net35.7% 1050.
1060. 1050. 1070.   7.3
11. cr83.la2ca.ip.att.net   28.6% 1049.
1064. 1049. 1072.   7.8
12. gar7.la2ca.ip.att.net   21.4% 1048.
1061. 1048. 1072.   6.7
13. 12.249.143.98   28.6% 1050.
1061. 1050. 1072.   7.9
14. ???

Just wanted to share in case anyone else is running into similar issues. I
know, I should be on the outages list. I will add myself now. :)

Regards,
Dave Sotnick