Re: Level 3 voice outage

[voytek]

Can anyone who was affected by last week's outage confirm that 911
services were impacted (I assume they were)?

Anyone know if the current outage is in any way related to this one from
last week?
http://downdetector.com/status/level3/map/



On 10/05/2016 01:24 PM, Mel Beckman wrote:
> It’s good to see them acknowledging this.
>
>  -mel
>
> On Oct 5, 2016, at 10:10 AM, Gareth Tupper 
> <gareth.tup...@warnerpacific.com<mailto:gareth.tup...@warnerpacific.com>> 
> wrote:
>
> Looks like a fat finger event...
>
> From Level 3:
> "On October 4, our voice network experienced a service disruption affecting 
> some of our customers in North America due to a configuration error. We know 
> how important these services are to our customers. As an organization, we're 
> putting processes in place to prevent issues like this from recurring in the 
> future. We were able to restore all services by 9:31am Mountain time."
>
>
> http://www.theregister.co.uk/2016/10/05/level3_voip_blackout_cause/
>
>
>
> -Original Message-
> From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Mel Beckman
> Sent: Tuesday, October 4, 2016 1:09 PM
> To: Marco Teixeira <ad...@marcoteixeira.com<mailto:ad...@marcoteixeira.com>>
> Cc: Shawn Ritchie <shawnritc...@gmail.com<mailto:shawnritc...@gmail.com>>; 
> NANOG list <nanog@nanog.org<mailto:nanog@nanog.org>>
> Subject: Re: Level 3 voice outage
>
> Possibly somebody YANGed when they should have yinged :)
>
> -mel beckman
>
> On Oct 4, 2016, at 1:06 PM, Marco Teixeira 
> <ad...@marcoteixeira.com<mailto:ad...@marcoteixeira.com><mailto:ad...@marcoteixeira.com>>
>  wrote:
>
> Yeap, i know, it was what i understood, as it is my opinion that a zero day 
> would fit better... in the pure speculation world :) At the end of the day... 
> maybe some undocumented fault int some obscure functionality that was 
> activated/deployed a long time ago, and just revealed it self now... There 
> are so many things that can go wrong on complex networks even with all the 
> controls imposed on changes...
>
>
>
>
> On Tue, Oct 4, 2016 at 8:54 PM, Shawn Ritchie 
> <shawnritc...@gmail.com<mailto:shawnritc...@gmail.com><mailto:shawnritc...@gmail.com>>
>  wrote:
> Well, Level3 has by no means said that this was the result of a DDoS, that's 
> just speculation on behalf of folks who do not work at Level3 so far.
>
> On Tue, Oct 4, 2016 at 2:49 PM Marco Teixeira 
> <ad...@marcoteixeira.com<mailto:ad...@marcoteixeira.com><mailto:ad...@marcoteixeira.com>>
>  wrote:
> I won't believe a company like Level3 would not deploy backplane 
> protection/policing on routers. Also, 1Tb/s aggregated DDoS towards OVH 
> network didn't pause or rebooted routers. And i guess both companies have had 
> their share of (D)DoS in the past, so they had the time to get up to the 
> challenge. Now... there where times where one malformed IP packet would cause 
> a memory leak leading to a router reboot... :)?
>
>
>
>
> On Tue, Oct 4, 2016 at 8:23 PM, Mel Beckman 
> <m...@beckman.org<mailto:m...@beckman.org><mailto:m...@beckman.org>> wrote:
>
> 765 Gbps per second directed at a router's interface IP might give the
> router pause, so to speak :)
>
> -mel
>
> On Oct 4, 2016, at 12:10 PM, Marco Teixeira
> <ad...@marcoteixeira.com<mailto:ad...@marcoteixeira.com><mailto:ad...@marcoteixeira.com>>
> wrote:
>
> Multiple reboots across several markets... Does not seem something
> that full pipes would trigger. Had it been an approved chance it would
> have been rolled back i guess... On the other hand, a zero day could apply...
>
> Em 04/10/2016 19:54, "Mel Beckman" 
> <m...@beckman.org<mailto:m...@beckman.org><mailto:m...@beckman.org>> escreveu:
>
> Sure. The recent release of the IoT DDoS attack code in the wild.
>
> -mel
>
> On Oct 4, 2016, at 11:42 AM, 
> valdis.kletni...@vt.edu<mailto:valdis.kletni...@vt.edu><mailto:valdis.kletni...@vt.edu>
>  wrote:
>
> On Tue, 04 Oct 2016 18:14:54 -, Mel Beckman said:
>
> This could be DoS attack.
>
> Or a missing comma in a code update.
>
> Or a fumble-fingered NOC monkey.
>
> Or
>
> You have any reason to suspect a DoS attack rather than all the
> other possibilities?
>
>
>
> --
>
> --
> Shawn
>
>
>
> This electronic mail transmission contains information from Warner Pacific 
> Insurance Services that may be confidential or privileged. Such information 
> is solely for the intended recipient, and use by any other party is not 
> authorized. If you are not the intended recipient, be aware that any 
> disclosure, copying, distribution or use of this message, its contents or any 
> attachments is prohibited. Any wrongful interception of this message is 
> punishable as a Federal Crime. If you have received this message in error, 
> please notify the sender immediately by telephone (800) 801-2300 or by 
> electronic mail at 
> postmas...@warnerpacific.com<mailto:postmas...@warnerpacific.com>.
>

Re: Level 3 voice outage

[Mel Beckman]

It’s good to see them acknowledging this.

 -mel

On Oct 5, 2016, at 10:10 AM, Gareth Tupper 
<gareth.tup...@warnerpacific.com<mailto:gareth.tup...@warnerpacific.com>> wrote:

Looks like a fat finger event...

From Level 3:
"On October 4, our voice network experienced a service disruption affecting 
some of our customers in North America due to a configuration error. We know 
how important these services are to our customers. As an organization, we're 
putting processes in place to prevent issues like this from recurring in the 
future. We were able to restore all services by 9:31am Mountain time."


http://www.theregister.co.uk/2016/10/05/level3_voip_blackout_cause/



-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Mel Beckman
Sent: Tuesday, October 4, 2016 1:09 PM
To: Marco Teixeira <ad...@marcoteixeira.com<mailto:ad...@marcoteixeira.com>>
Cc: Shawn Ritchie <shawnritc...@gmail.com<mailto:shawnritc...@gmail.com>>; 
NANOG list <nanog@nanog.org<mailto:nanog@nanog.org>>
Subject: Re: Level 3 voice outage

Possibly somebody YANGed when they should have yinged :)

-mel beckman

On Oct 4, 2016, at 1:06 PM, Marco Teixeira 
<ad...@marcoteixeira.com<mailto:ad...@marcoteixeira.com><mailto:ad...@marcoteixeira.com>>
 wrote:

Yeap, i know, it was what i understood, as it is my opinion that a zero day 
would fit better... in the pure speculation world :) At the end of the day... 
maybe some undocumented fault int some obscure functionality that was 
activated/deployed a long time ago, and just revealed it self now... There are 
so many things that can go wrong on complex networks even with all the controls 
imposed on changes...




On Tue, Oct 4, 2016 at 8:54 PM, Shawn Ritchie 
<shawnritc...@gmail.com<mailto:shawnritc...@gmail.com><mailto:shawnritc...@gmail.com>>
 wrote:
Well, Level3 has by no means said that this was the result of a DDoS, that's 
just speculation on behalf of folks who do not work at Level3 so far.

On Tue, Oct 4, 2016 at 2:49 PM Marco Teixeira 
<ad...@marcoteixeira.com<mailto:ad...@marcoteixeira.com><mailto:ad...@marcoteixeira.com>>
 wrote:
I won't believe a company like Level3 would not deploy backplane 
protection/policing on routers. Also, 1Tb/s aggregated DDoS towards OVH network 
didn't pause or rebooted routers. And i guess both companies have had their 
share of (D)DoS in the past, so they had the time to get up to the challenge. 
Now... there where times where one malformed IP packet would cause a memory 
leak leading to a router reboot... :)?




On Tue, Oct 4, 2016 at 8:23 PM, Mel Beckman 
<m...@beckman.org<mailto:m...@beckman.org><mailto:m...@beckman.org>> wrote:

765 Gbps per second directed at a router's interface IP might give the
router pause, so to speak :)

-mel

On Oct 4, 2016, at 12:10 PM, Marco Teixeira
<ad...@marcoteixeira.com<mailto:ad...@marcoteixeira.com><mailto:ad...@marcoteixeira.com>>
wrote:

Multiple reboots across several markets... Does not seem something
that full pipes would trigger. Had it been an approved chance it would
have been rolled back i guess... On the other hand, a zero day could apply...

Em 04/10/2016 19:54, "Mel Beckman" 
<m...@beckman.org<mailto:m...@beckman.org><mailto:m...@beckman.org>> escreveu:

Sure. The recent release of the IoT DDoS attack code in the wild.

-mel

On Oct 4, 2016, at 11:42 AM, 
valdis.kletni...@vt.edu<mailto:valdis.kletni...@vt.edu><mailto:valdis.kletni...@vt.edu>
 wrote:

On Tue, 04 Oct 2016 18:14:54 -, Mel Beckman said:

This could be DoS attack.

Or a missing comma in a code update.

Or a fumble-fingered NOC monkey.

Or

You have any reason to suspect a DoS attack rather than all the
other possibilities?



--

--
Shawn



This electronic mail transmission contains information from Warner Pacific 
Insurance Services that may be confidential or privileged. Such information is 
solely for the intended recipient, and use by any other party is not 
authorized. If you are not the intended recipient, be aware that any 
disclosure, copying, distribution or use of this message, its contents or any 
attachments is prohibited. Any wrongful interception of this message is 
punishable as a Federal Crime. If you have received this message in error, 
please notify the sender immediately by telephone (800) 801-2300 or by 
electronic mail at 
postmas...@warnerpacific.com<mailto:postmas...@warnerpacific.com>.

RE: Level 3 voice outage

[Gareth Tupper]

Looks like a fat finger event...

>From Level 3:
"On October 4, our voice network experienced a service disruption affecting 
some of our customers in North America due to a configuration error. We know 
how important these services are to our customers. As an organization, we're 
putting processes in place to prevent issues like this from recurring in the 
future. We were able to restore all services by 9:31am Mountain time."


http://www.theregister.co.uk/2016/10/05/level3_voip_blackout_cause/



-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Mel Beckman
Sent: Tuesday, October 4, 2016 1:09 PM
To: Marco Teixeira <ad...@marcoteixeira.com>
Cc: Shawn Ritchie <shawnritc...@gmail.com>; NANOG list <nanog@nanog.org>
Subject: Re: Level 3 voice outage

Possibly somebody YANGed when they should have yinged :)

 -mel beckman

On Oct 4, 2016, at 1:06 PM, Marco Teixeira 
<ad...@marcoteixeira.com<mailto:ad...@marcoteixeira.com>> wrote:

Yeap, i know, it was what i understood, as it is my opinion that a zero day 
would fit better... in the pure speculation world :) At the end of the day... 
maybe some undocumented fault int some obscure functionality that was 
activated/deployed a long time ago, and just revealed it self now... There are 
so many things that can go wrong on complex networks even with all the controls 
imposed on changes...




On Tue, Oct 4, 2016 at 8:54 PM, Shawn Ritchie 
<shawnritc...@gmail.com<mailto:shawnritc...@gmail.com>> wrote:
Well, Level3 has by no means said that this was the result of a DDoS, that's 
just speculation on behalf of folks who do not work at Level3 so far.

On Tue, Oct 4, 2016 at 2:49 PM Marco Teixeira 
<ad...@marcoteixeira.com<mailto:ad...@marcoteixeira.com>> wrote:
I won't believe a company like Level3 would not deploy backplane 
protection/policing on routers. Also, 1Tb/s aggregated DDoS towards OVH network 
didn't pause or rebooted routers. And i guess both companies have had their 
share of (D)DoS in the past, so they had the time to get up to the challenge. 
Now... there where times where one malformed IP packet would cause a memory 
leak leading to a router reboot... :)?




On Tue, Oct 4, 2016 at 8:23 PM, Mel Beckman 
<m...@beckman.org<mailto:m...@beckman.org>> wrote:

> 765 Gbps per second directed at a router's interface IP might give the
> router pause, so to speak :)
>
>  -mel
>
> On Oct 4, 2016, at 12:10 PM, Marco Teixeira
> <ad...@marcoteixeira.com<mailto:ad...@marcoteixeira.com>>
> wrote:
>
> Multiple reboots across several markets... Does not seem something
> that full pipes would trigger. Had it been an approved chance it would
> have been rolled back i guess... On the other hand, a zero day could apply...
>
> Em 04/10/2016 19:54, "Mel Beckman" 
> <m...@beckman.org<mailto:m...@beckman.org>> escreveu:
>
>> Sure. The recent release of the IoT DDoS attack code in the wild.
>>
>>  -mel
>>
>> > On Oct 4, 2016, at 11:42 AM, 
>> > valdis.kletni...@vt.edu<mailto:valdis.kletni...@vt.edu> wrote:
>> >
>> > On Tue, 04 Oct 2016 18:14:54 -, Mel Beckman said:
>> >
>> >> This could be DoS attack.
>> >
>> > Or a missing comma in a code update.
>> >
>> > Or a fumble-fingered NOC monkey.
>> >
>> > Or
>> >
>> > You have any reason to suspect a DoS attack rather than all the
>> > other possibilities?
>>
>>
>
--

--
Shawn



This electronic mail transmission contains information from Warner Pacific 
Insurance Services that may be confidential or privileged. Such information is 
solely for the intended recipient, and use by any other party is not 
authorized. If you are not the intended recipient, be aware that any 
disclosure, copying, distribution or use of this message, its contents or any 
attachments is prohibited. Any wrongful interception of this message is 
punishable as a Federal Crime. If you have received this message in error, 
please notify the sender immediately by telephone (800) 801-2300 or by 
electronic mail at postmas...@warnerpacific.com.

Re: Level 3 voice outage

[Mel Beckman]

Possibly somebody YANGed when they should have yinged :)

 -mel beckman

On Oct 4, 2016, at 1:06 PM, Marco Teixeira 
> wrote:

Yeap, i know, it was what i understood, as it is my opinion that a zero day 
would fit better... in the pure speculation world :)
At the end of the day... maybe some undocumented fault int some obscure 
functionality that was activated/deployed a long time ago, and just revealed it 
self now... There are so many things that can go wrong on complex networks even 
with all the controls imposed on changes...




On Tue, Oct 4, 2016 at 8:54 PM, Shawn Ritchie 
> wrote:
Well, Level3 has by no means said that this was the result of a DDoS, that's 
just speculation on behalf of folks who do not work at Level3 so far.

On Tue, Oct 4, 2016 at 2:49 PM Marco Teixeira 
> wrote:
I won't believe a company like Level3 would not deploy backplane
protection/policing on routers. Also, 1Tb/s aggregated DDoS towards OVH
network didn't pause or rebooted routers. And i guess both companies have
had their share of (D)DoS in the past, so they had the time to get up to
the challenge. Now... there where times where one malformed IP packet would
cause a memory leak leading to a router reboot... :)?




On Tue, Oct 4, 2016 at 8:23 PM, Mel Beckman 
> wrote:

> 765 Gbps per second directed at a router's interface IP might give the
> router pause, so to speak :)
>
>  -mel
>
> On Oct 4, 2016, at 12:10 PM, Marco Teixeira 
> >
> wrote:
>
> Multiple reboots across several markets... Does not seem something that
> full pipes would trigger. Had it been an approved chance it would have been
> rolled back i guess... On the other hand, a zero day could apply...
>
> Em 04/10/2016 19:54, "Mel Beckman" 
> > escreveu:
>
>> Sure. The recent release of the IoT DDoS attack code in the wild.
>>
>>  -mel
>>
>> > On Oct 4, 2016, at 11:42 AM, 
>> > valdis.kletni...@vt.edu wrote:
>> >
>> > On Tue, 04 Oct 2016 18:14:54 -, Mel Beckman said:
>> >
>> >> This could be DoS attack.
>> >
>> > Or a missing comma in a code update.
>> >
>> > Or a fumble-fingered NOC monkey.
>> >
>> > Or
>> >
>> > You have any reason to suspect a DoS attack rather than all the other
>> > possibilities?
>>
>>
>
--

--
Shawn

Re: Level 3 voice outage

[Marco Teixeira]

Yeap, i know, it was what i understood, as it is my opinion that a zero day
would fit better... in the pure speculation world :)
At the end of the day... maybe some undocumented fault int some obscure
functionality that was activated/deployed a long time ago, and just
revealed it self now... There are so many things that can go wrong on
complex networks even with all the controls imposed on changes...




On Tue, Oct 4, 2016 at 8:54 PM, Shawn Ritchie 
wrote:

> Well, Level3 has by no means said that this was the result of a DDoS,
> that's just speculation on behalf of folks who do not work at Level3 so
> far.
>
> On Tue, Oct 4, 2016 at 2:49 PM Marco Teixeira 
> wrote:
>
>> I won't believe a company like Level3 would not deploy backplane
>> protection/policing on routers. Also, 1Tb/s aggregated DDoS towards OVH
>> network didn't pause or rebooted routers. And i guess both companies have
>> had their share of (D)DoS in the past, so they had the time to get up to
>> the challenge. Now... there where times where one malformed IP packet
>> would
>> cause a memory leak leading to a router reboot... :)​
>>
>>
>>
>>
>> On Tue, Oct 4, 2016 at 8:23 PM, Mel Beckman  wrote:
>>
>> > 765 Gbps per second directed at a router’s interface IP might give the
>> > router pause, so to speak :)
>> >
>> >  -mel
>> >
>> > On Oct 4, 2016, at 12:10 PM, Marco Teixeira 
>> > wrote:
>> >
>> > Multiple reboots across several markets... Does not seem something that
>> > full pipes would trigger. Had it been an approved chance it would have
>> been
>> > rolled back i guess... On the other hand, a zero day could apply...
>> >
>> > Em 04/10/2016 19:54, "Mel Beckman"  escreveu:
>> >
>> >> Sure. The recent release of the IoT DDoS attack code in the wild.
>> >>
>> >>  -mel
>> >>
>> >> > On Oct 4, 2016, at 11:42 AM, valdis.kletni...@vt.edu wrote:
>> >> >
>> >> > On Tue, 04 Oct 2016 18:14:54 -, Mel Beckman said:
>> >> >
>> >> >> This could be DoS attack.
>> >> >
>> >> > Or a missing comma in a code update.
>> >> >
>> >> > Or a fumble-fingered NOC monkey.
>> >> >
>> >> > Or
>> >> >
>> >> > You have any reason to suspect a DoS attack rather than all the other
>> >> > possibilities?
>> >>
>> >>
>> >
>>
> --
>
> --
> Shawn
>

Re: Level 3 voice outage

[Mel Beckman]

Just so. We are just speculating, but in light of recent GRE attack 
developments, the ddos possibility is worth considering.

 -mel beckman

On Oct 4, 2016, at 12:54 PM, Shawn Ritchie 
> wrote:

Well, Level3 has by no means said that this was the result of a DDoS, that's 
just speculation on behalf of folks who do not work at Level3 so far.

On Tue, Oct 4, 2016 at 2:49 PM Marco Teixeira 
> wrote:
I won't believe a company like Level3 would not deploy backplane
protection/policing on routers. Also, 1Tb/s aggregated DDoS towards OVH
network didn't pause or rebooted routers. And i guess both companies have
had their share of (D)DoS in the past, so they had the time to get up to
the challenge. Now... there where times where one malformed IP packet would
cause a memory leak leading to a router reboot... :)?




On Tue, Oct 4, 2016 at 8:23 PM, Mel Beckman 
> wrote:

> 765 Gbps per second directed at a router's interface IP might give the
> router pause, so to speak :)
>
>  -mel
>
> On Oct 4, 2016, at 12:10 PM, Marco Teixeira 
> >
> wrote:
>
> Multiple reboots across several markets... Does not seem something that
> full pipes would trigger. Had it been an approved chance it would have been
> rolled back i guess... On the other hand, a zero day could apply...
>
> Em 04/10/2016 19:54, "Mel Beckman" 
> > escreveu:
>
>> Sure. The recent release of the IoT DDoS attack code in the wild.
>>
>>  -mel
>>
>> > On Oct 4, 2016, at 11:42 AM, 
>> > valdis.kletni...@vt.edu wrote:
>> >
>> > On Tue, 04 Oct 2016 18:14:54 -, Mel Beckman said:
>> >
>> >> This could be DoS attack.
>> >
>> > Or a missing comma in a code update.
>> >
>> > Or a fumble-fingered NOC monkey.
>> >
>> > Or
>> >
>> > You have any reason to suspect a DoS attack rather than all the other
>> > possibilities?
>>
>>
>
--

--
Shawn

Re: Level 3 voice outage

[Marco Teixeira]

... but back then they didn't have ITIL and the likes (at least not that i
knew... but then again, i was just starting with networking back then),
where a ChM process must produce risk, impact, etc, reports, and at least
one peer review meeting for approval :)

=
Marco

On Tue, Oct 4, 2016 at 8:33 PM,  wrote:

> On Tue, 04 Oct 2016 20:10:44 +0100, Marco Teixeira said:
> > Had it been an approved chance it would have been
> > rolled back i guess...
>
> See the 1990 ATT long-distance collapse for a worked example.
>

Re: Level 3 voice outage

[Marco Teixeira]

I won't believe a company like Level3 would not deploy backplane
protection/policing on routers. Also, 1Tb/s aggregated DDoS towards OVH
network didn't pause or rebooted routers. And i guess both companies have
had their share of (D)DoS in the past, so they had the time to get up to
the challenge. Now... there where times where one malformed IP packet would
cause a memory leak leading to a router reboot... :)​




On Tue, Oct 4, 2016 at 8:23 PM, Mel Beckman  wrote:

> 765 Gbps per second directed at a router’s interface IP might give the
> router pause, so to speak :)
>
>  -mel
>
> On Oct 4, 2016, at 12:10 PM, Marco Teixeira 
> wrote:
>
> Multiple reboots across several markets... Does not seem something that
> full pipes would trigger. Had it been an approved chance it would have been
> rolled back i guess... On the other hand, a zero day could apply...
>
> Em 04/10/2016 19:54, "Mel Beckman"  escreveu:
>
>> Sure. The recent release of the IoT DDoS attack code in the wild.
>>
>>  -mel
>>
>> > On Oct 4, 2016, at 11:42 AM, valdis.kletni...@vt.edu wrote:
>> >
>> > On Tue, 04 Oct 2016 18:14:54 -, Mel Beckman said:
>> >
>> >> This could be DoS attack.
>> >
>> > Or a missing comma in a code update.
>> >
>> > Or a fumble-fingered NOC monkey.
>> >
>> > Or
>> >
>> > You have any reason to suspect a DoS attack rather than all the other
>> > possibilities?
>>
>>
>

Re: Level 3 voice outage

[Valdis . Kletnieks]

On Tue, 04 Oct 2016 20:10:44 +0100, Marco Teixeira said:
> Had it been an approved chance it would have been
> rolled back i guess...

See the 1990 ATT long-distance collapse for a worked example.


pgp_Sro2vzM5h.pgp
Description: PGP signature

Re: Level 3 voice outage

[Mel Beckman]

765 Gbps per second directed at a router’s interface IP might give the router 
pause, so to speak :)

 -mel

On Oct 4, 2016, at 12:10 PM, Marco Teixeira 
> wrote:


Multiple reboots across several markets... Does not seem something that full 
pipes would trigger. Had it been an approved chance it would have been rolled 
back i guess... On the other hand, a zero day could apply...

Em 04/10/2016 19:54, "Mel Beckman" > 
escreveu:
Sure. The recent release of the IoT DDoS attack code in the wild.

 -mel

> On Oct 4, 2016, at 11:42 AM, 
> valdis.kletni...@vt.edu wrote:
>
> On Tue, 04 Oct 2016 18:14:54 -, Mel Beckman said:
>
>> This could be DoS attack.
>
> Or a missing comma in a code update.
>
> Or a fumble-fingered NOC monkey.
>
> Or
>
> You have any reason to suspect a DoS attack rather than all the other
> possibilities?

Re: Level 3 voice outage

[Marco Teixeira]

Multiple reboots across several markets... Does not seem something that
full pipes would trigger. Had it been an approved chance it would have been
rolled back i guess... On the other hand, a zero day could apply...

Em 04/10/2016 19:54, "Mel Beckman"  escreveu:

> Sure. The recent release of the IoT DDoS attack code in the wild.
>
>  -mel
>
> > On Oct 4, 2016, at 11:42 AM, valdis.kletni...@vt.edu wrote:
> >
> > On Tue, 04 Oct 2016 18:14:54 -, Mel Beckman said:
> >
> >> This could be DoS attack.
> >
> > Or a missing comma in a code update.
> >
> > Or a fumble-fingered NOC monkey.
> >
> > Or
> >
> > You have any reason to suspect a DoS attack rather than all the other
> > possibilities?
>
>

Re: Level 3 voice outage

[Mel Beckman]

Sure. The recent release of the IoT DDoS attack code in the wild.

 -mel

> On Oct 4, 2016, at 11:42 AM, valdis.kletni...@vt.edu wrote:
> 
> On Tue, 04 Oct 2016 18:14:54 -, Mel Beckman said:
> 
>> This could be DoS attack.
> 
> Or a missing comma in a code update.
> 
> Or a fumble-fingered NOC monkey.
> 
> Or
> 
> You have any reason to suspect a DoS attack rather than all the other
> possibilities?

Re: Level 3 voice outage

[Valdis . Kletnieks]

On Tue, 04 Oct 2016 18:14:54 -, Mel Beckman said:

> This could be DoS attack.

Or a missing comma in a code update.

Or a fumble-fingered NOC monkey.

Or

You have any reason to suspect a DoS attack rather than all the other
possibilities?


pgp03Ijae5BSG.pgp
Description: PGP signature

Re: Level 3 voice outage

[Mel Beckman]

We are seeing high latencies and loss of transit through Level3 in Southern 
California. 

Level 3 has opened Global Ticket 115724: Network devices in multiple markets in 
North America are rebooting, impacting IP services.”

This could be DoS attack.


> On Oct 4, 2016, at 11:06 AM, Jason Hellenthal  wrote:
> 
> Patience Obi Wan ! They are investigating the root cause and like most root 
> causes they don’t just hold out a flag and say here I am !!!
> 
>> On Oct 4, 2016, at 10:32, Ivaylo Katovski  wrote:
>> 
>> When will L3 notify their customers for the outage?!? According to l3
>> twitter account their are aware of the voice impact and working on it
>> 
>> On Oct 4, 2016 11:03 AM, "Mark Stevens"  wrote:
>> 
>>> Is anyone noticing issue with Level 3 voice? I can't even call their 800
>>> number using one of my other carriers.
>>> 
>>> Mark
>>> 
> 
> 
> -- 
> Jason Hellenthal
> JJH48-ARIN
> 
> 
> 
> 

Re: Level 3 voice outage

[Jason Hellenthal]

Patience Obi Wan ! They are investigating the root cause and like most root 
causes they don’t just hold out a flag and say here I am !!!

> On Oct 4, 2016, at 10:32, Ivaylo Katovski  wrote:
> 
> When will L3 notify their customers for the outage?!? According to l3
> twitter account their are aware of the voice impact and working on it
> 
> On Oct 4, 2016 11:03 AM, "Mark Stevens"  wrote:
> 
>> Is anyone noticing issue with Level 3 voice? I can't even call their 800
>> number using one of my other carriers.
>> 
>> Mark
>> 


-- 
 Jason Hellenthal
 JJH48-ARIN

Re: Level 3 voice outage

[Ivaylo Katovski]

When will L3 notify their customers for the outage?!? According to l3
twitter account their are aware of the voice impact and working on it

On Oct 4, 2016 11:03 AM, "Mark Stevens"  wrote:

> Is anyone noticing issue with Level 3 voice? I can't even call their 800
> number using one of my other carriers.
>
> Mark
>

RE: [BULK] Re: Level 3 voice outage

[Asche, Ron]

Boise Idaho is affected...

-Original Message-
From: NANOG [mailto:nanog-bounces+ascher=slhs@nanog.org] On Behalf Of 
Benjamin Hatton
Sent: Tuesday, October 04, 2016 9:04 AM
To: nanog@nanog.org
Subject: [BULK] Re: Level 3 voice outage
Importance: High

WARNING: This email originated outside of St. Luke’s email system.
DO NOT CLICK links or attachments unless you recognize the sender and know the 
content is safe.


We just got a notification from one of our upstream Voice providers that they 
are seeing problems both inbound and outbound that seems to be Level 3 related

*Ben Hatton*

Network Engineer

Haefele TV Inc.

d:(607)589-8000

bhat...@htva.net

www.htva.net

On Tue, Oct 4, 2016 at 11:01 AM, Mark Stevens <mana...@monmouth.com> wrote:

> Is anyone noticing issue with Level 3 voice? I can't even call their 
> 800 number using one of my other carriers.
>
> Mark
>


"This message is intended for the use of the person or entity to which it is 
addressed and may contain information that is confidential or privileged, the 
disclosure of which is governed by applicable law. If the reader of this 
message is not the intended recipient, you are hereby notified that any 
dissemination, distribution, or copying of this information is strictly 
prohibited. If you have received this message by error, please notify us 
immediately and destroy the related message." 

RE: Level 3 voice outage

[Ken Warnimont]

There's all sorts of people online reporting problems with Level3 voice 
services - we're down for outbound calling, inbound is spotty.  Their portal is 
unreachable, can't get to any of their #'s...seems like the world is on fire 
over there.

We were a previous TW Telecom customer before Level3 bought them out, the 
service quality has definitely degraded since then.

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Mark Stevens
Sent: Tuesday, October 04, 2016 10:02 AM
To: nanog@nanog.org
Subject: Level 3 voice outage

Is anyone noticing issue with Level 3 voice? I can't even call their 800 number 
using one of my other carriers.

Mark

***
This email message, including any attachments, is intended solely for the 
designated person or entity to which the message is addressed, and may contain 
confidential information. It is strictly prohibited to review, distribute, 
copy, or print this email, including attachments, by any person or entity other 
than the designated addressee. 
If you have erroneously received this message, please contact the sender 
immediately and remove the data from any computer on which the message resides.

Re: Level 3 voice outage

[Shane Heupel]

We can't make any LD calls through Level3 either. 

Thank you,
Shane Heupel

> On Oct 4, 2016, at 10:06 AM, Benjamin Hatton  wrote:
> 
> We just got a notification from one of our upstream Voice providers that
> they are seeing problems both inbound and outbound that seems to be Level 3
> related
> 
> *Ben Hatton*
> 
> Network Engineer
> 
> Haefele TV Inc.
> 
> d:(607)589-8000
> 
> bhat...@htva.net
> 
> www.htva.net
> 
>> On Tue, Oct 4, 2016 at 11:01 AM, Mark Stevens  wrote:
>> 
>> Is anyone noticing issue with Level 3 voice? I can't even call their 800
>> number using one of my other carriers.
>> 
>> Mark
>> 

Re: Level 3 voice outage

[Mark Horton]

Major Level 3 outage; they are currently not taking support calls.








On 10/4/16, 8:01 AM, "NANOG on behalf of Mark Stevens"  wrote:

>Is anyone noticing issue with Level 3 voice? I can't even call their 800 
>number using one of my other carriers.
>
>Mark

Re: Level 3 voice outage

[C. Jon Larsen]

On Tue, 4 Oct 2016, Mark Stevens wrote:

Is anyone noticing issue with Level 3 voice? I can't even call their 800 
number using one of my other carriers.


Yes, major outage:
http://downdetector.com/status/level3

Customers all over the mid-atlantic region are down.

RE: Level 3 voice outage

[timrutherford]

We are back in action here in MA.

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of timrutherf...@c4.net
Sent: Tuesday, October 4, 2016 11:20 AM
To: nanog@nanog.org
Subject: RE: Level 3 voice outage

Our upstream Voice provider is also having inbound issues with numbers that 
were ported into their system, but the main trunk number is working.

-Tim

-Original Message-
From: NANOG [mailto:nanog-bounces+timrutherford=c4@nanog.org] On Behalf Of 
Benjamin Hatton
Sent: Tuesday, October 4, 2016 11:04 AM
To: nanog@nanog.org
Subject: Re: Level 3 voice outage

We just got a notification from one of our upstream Voice providers that they 
are seeing problems both inbound and outbound that seems to be Level 3 related

*Ben Hatton*

Network Engineer

Haefele TV Inc.

d:(607)589-8000

bhat...@htva.net

www.htva.net

On Tue, Oct 4, 2016 at 11:01 AM, Mark Stevens <mana...@monmouth.com> wrote:

> Is anyone noticing issue with Level 3 voice? I can't even call their 
> 800 number using one of my other carriers.
>
> Mark
>

Re: Level 3 voice outage

[Sean]

Twitter is blowing up ( https://twitter.com/hashtag/outage ) and their
website (www.level3.com) is down for me so yeah... I'd say they're having a
frowny face day.

On Tue, Oct 4, 2016 at 10:20 AM, <timrutherf...@c4.net> wrote:

> Our upstream Voice provider is also having inbound issues with numbers
> that were ported into their system, but the main trunk number is working.
>
> -Tim
>
> -Original Message-
> From: NANOG [mailto:nanog-bounces+timrutherford=c4@nanog.org] On
> Behalf Of Benjamin Hatton
> Sent: Tuesday, October 4, 2016 11:04 AM
> To: nanog@nanog.org
> Subject: Re: Level 3 voice outage
>
> We just got a notification from one of our upstream Voice providers that
> they are seeing problems both inbound and outbound that seems to be Level 3
> related
>
> *Ben Hatton*
>
> Network Engineer
>
> Haefele TV Inc.
>
> d:(607)589-8000
>
> bhat...@htva.net
>
> www.htva.net
>
> On Tue, Oct 4, 2016 at 11:01 AM, Mark Stevens <mana...@monmouth.com>
> wrote:
>
> > Is anyone noticing issue with Level 3 voice? I can't even call their
> > 800 number using one of my other carriers.
> >
> > Mark
> >
>
>
>

Re: Level 3 voice outage

[Adair Winter]

Our provider is down as well. Saying Level 3 issues. Anyone know what's
going on or and ETR?

On Tue, Oct 4, 2016 at 10:06 AM, Kevin Kadow  wrote:

> Level 3 has issues nationwide today.  We have voice outages at our offices
> in multiple cities, Chicago and out west.
>
> On Tue, Oct 4, 2016 at 11:01 AM, Mark Stevens 
> wrote:
>
> > Is anyone noticing issue with Level 3 voice? I can't even call their 800
> > number using one of my other carriers.
> >
> > Mark
> >
>



-- 

Adair Winter
VP, Network Operations / Co-Owner
Amarillo Wireless | 806.316.5071
C: 806.231.7180
http://www.amarillowireless.net

RE: Level 3 voice outage

[timrutherford]

Our upstream Voice provider is also having inbound issues with numbers that 
were ported into their system, but the main trunk number is working.

-Tim

-Original Message-
From: NANOG [mailto:nanog-bounces+timrutherford=c4@nanog.org] On Behalf Of 
Benjamin Hatton
Sent: Tuesday, October 4, 2016 11:04 AM
To: nanog@nanog.org
Subject: Re: Level 3 voice outage

We just got a notification from one of our upstream Voice providers that they 
are seeing problems both inbound and outbound that seems to be Level 3 related

*Ben Hatton*

Network Engineer

Haefele TV Inc.

d:(607)589-8000

bhat...@htva.net

www.htva.net

On Tue, Oct 4, 2016 at 11:01 AM, Mark Stevens <mana...@monmouth.com> wrote:

> Is anyone noticing issue with Level 3 voice? I can't even call their 
> 800 number using one of my other carriers.
>
> Mark
>

Re: Level 3 voice outage

[Matt Freitag]

There's a long thread over at the outages list. Sign-up for the outages
list is at https://puck.nether.net/mailman/listinfo/outages and the archive
for the whole thread starts at
https://puck.nether.net/pipermail/outages/2016-October/009609.html.

Matt Freitag
Network Engineer I
Information Technology
Michigan Technological University
(906) 487-3696 <%28906%29%20487-3696>
https://www.mtu.edu/
https://www.it.mtu.edu/

On Tue, Oct 4, 2016 at 11:06 AM, Kevin Kadow  wrote:

> Level 3 has issues nationwide today.  We have voice outages at our offices
> in multiple cities, Chicago and out west.
>
> On Tue, Oct 4, 2016 at 11:01 AM, Mark Stevens 
> wrote:
>
> > Is anyone noticing issue with Level 3 voice? I can't even call their 800
> > number using one of my other carriers.
> >
> > Mark
> >
>

Re: Level 3 voice outage

[Kevin Kadow]

Level 3 has issues nationwide today.  We have voice outages at our offices
in multiple cities, Chicago and out west.

On Tue, Oct 4, 2016 at 11:01 AM, Mark Stevens  wrote:

> Is anyone noticing issue with Level 3 voice? I can't even call their 800
> number using one of my other carriers.
>
> Mark
>

Re: Level 3 voice outage

[Benjamin Hatton]

We just got a notification from one of our upstream Voice providers that
they are seeing problems both inbound and outbound that seems to be Level 3
related

*Ben Hatton*

Network Engineer

Haefele TV Inc.

d:(607)589-8000

bhat...@htva.net

www.htva.net

On Tue, Oct 4, 2016 at 11:01 AM, Mark Stevens  wrote:

> Is anyone noticing issue with Level 3 voice? I can't even call their 800
> number using one of my other carriers.
>
> Mark
>

Re: Level 3 voice outage?

[Ryan, Spencer]

Ran across this earlier, it sounds bad.


https://www.reddit.com/r/networking/comments/504xbo/level_3_voice_outage_global_ticket_being_worked/


Spencer Ryan | Senior Systems Administrator | 
sr...@arbor.net
Arbor Networks
+1.734.794.5033 (d) | +1.734.846.2053 (m)
www.arbornetworks.com



From: NANOG  on behalf of David Hubbard 

Sent: Monday, August 29, 2016 10:31:19 AM
To: nanog@nanog.org
Subject: Level 3 voice outage?

Curious if anyone else is having issues with Level 3 (legacy Twtelecom 
specifically) enterprise SIP?  I’m at minute 45 of being on hold with them, so 
I suspect they are having known issues.  Our sales rep mentioned a toll free 
outage being tracked under master ticket 11377637 but I don’t have the details 
of that yet.

We’re seeing our toll free numbers completely down, since what I believe to be 
4a EST time frame.  Most of our toll numbers have an unusual 25 second delay 
before we get any SIP traffic from their equipment, but the call does 
ultimately connect.

David