RE: Level3 worldwide emergency upgrade?

2013-02-10 Thread Simon Allard
; Jonathan Towne Cc: nanog@nanog.org Subject: RE: Level3 worldwide emergency upgrade? I think you might find its this issue. PSN-2013-01-823 "Junos: Crafted TCP packet can lead to kernel crash" -Original Message- From: Matthew Petach [mailto:mpet...@netflight.com] Sent: T

RE: Level3 worldwide emergency upgrade?

2013-02-10 Thread Simon Allard
Re: Level3 worldwide emergency upgrade? On Wed, Feb 6, 2013 at 5:10 AM, Jonathan Towne wrote: > On Wed, Feb 06, 2013 at 07:57:06AM -0500, Alex Rubenstein scribbled: > # The question should be more along the lines of, "why aren't you multihomed > in a way that would make a 30 minut

Re: Level3 worldwide emergency upgrade?

2013-02-07 Thread Dorian Kim
iginal Message- > From: Brett Watson [mailto:br...@the-watsons.org] > Sent: Wednesday, February 06, 2013 6:07 PM > To: nanog@nanog.org > Subject: Re: Level3 worldwide emergency upgrade? > > Hell, we used to not have to bother notifying customers of anything, we just > fixed the

RE: Level3 worldwide emergency upgrade?

2013-02-07 Thread Siegel, David
Brett Watson [mailto:br...@the-watsons.org] Sent: Wednesday, February 06, 2013 6:07 PM To: nanog@nanog.org Subject: Re: Level3 worldwide emergency upgrade? Hell, we used to not have to bother notifying customers of anything, we just fixed the problem. Reminds me a of a story I've probably sha

Re: Level3 worldwide emergency upgrade?

2013-02-07 Thread Brian Landers
The Juniper PR in question is actually 836197. On Wed, Feb 6, 2013 at 10:22 AM, Matthew Petach wrote: > On Wed, Feb 6, 2013 at 5:10 AM, Jonathan Towne wrote: > > On Wed, Feb 06, 2013 at 07:57:06AM -0500, Alex Rubenstein scribbled: > > # The question should be more along the lines of, "why aren'

Re: Level3 worldwide emergency upgrade?

2013-02-07 Thread Jeff Tantsura
Op Sec. >> >> When it is that, it is best if the remainder of us sit quietly on the >> sidelines. >> >> Ralph Brandt >> >> >> -----Original Message- >> From: Siegel, David [mailto:david.sie...@level3.com] >> Sent: Wednesday

Re: Level3 worldwide emergency upgrade?

2013-02-06 Thread bmanning
responsibility so it had to be Op Sec. > > > > When it is that, it is best if the remainder of us sit quietly on the > > sidelines. > > > > Ralph Brandt > > > > > > -Original Message- > > From: Siegel, David [mailto:david.sie...@level3.com]

Re: Level3 worldwide emergency upgrade?

2013-02-06 Thread Brett Watson
com] > Sent: Wednesday, February 06, 2013 12:01 PM > To: 'Ray Wong'; nanog@nanog.org > Subject: RE: Level3 worldwide emergency upgrade? > > Hi Ray, > > This topic reminds me of yesterday's discussion in the conference around > getting some BCOP's draft

Re: Level3 worldwide emergency upgrade?

2013-02-06 Thread joel jaeggli
To: 'Ray Wong'; nanog@nanog.org Subject: RE: Level3 worldwide emergency upgrade? Hi Ray, This topic reminds me of yesterday's discussion in the conference around getting some BCOP's drafted. it would be useful to confirm my own view of the BCOP around communicating

RE: Level3 worldwide emergency upgrade?

2013-02-06 Thread Brandt, Ralph
] Sent: Wednesday, February 06, 2013 12:01 PM To: 'Ray Wong'; nanog@nanog.org Subject: RE: Level3 worldwide emergency upgrade? Hi Ray, This topic reminds me of yesterday's discussion in the conference around getting some BCOP's drafted. it would be useful to confirm my own vie

Re: Level3 worldwide emergency upgrade?

2013-02-06 Thread Florian Weimer
* Andrew Sullivan: > My impression is mostly that people are left feeling uncomfortable > by a massive upgrade of this sort with so little communication about > why and so on. That's a side effect of Juniper's notification policy. Perhaps someone should them take them by their word ("Security pa

Re: Level3 worldwide emergency upgrade?

2013-02-06 Thread Alexander Maassen
On Wed, 2013-02-06 at 07:57 -0500, Alex Rubenstein wrote: > > Would you rather your ISP not maintain their devices? Are the > > consequences "so bad" of a 30 minute outage that your business > > is severely impacted? > > > > - Jared > > You had me up until that line. > > That should be expande

Re: Level3 worldwide emergency upgrade?

2013-02-06 Thread Matthew Petach
On Wed, Feb 6, 2013 at 5:10 AM, Jonathan Towne wrote: > On Wed, Feb 06, 2013 at 07:57:06AM -0500, Alex Rubenstein scribbled: > # The question should be more along the lines of, "why aren't you multihomed > in a way that would make a 30 minute outage (which is inevitable) irrelevant > to you? > >

Re: Level3 worldwide emergency upgrade?

2013-02-06 Thread joel jaeggli
On 2/6/13 8:34 AM, Justin M. Streiner wrote: On Wed, 6 Feb 2013, Ray Wong wrote: My impression is mostly that people are left feeling uncomfortable by a massive upgrade of this sort with so little communication about why and so on. "Emergency work for five hours and 30 minutes disconnection" t

RE: Level3 worldwide emergency upgrade?

2013-02-06 Thread Siegel, David
ubject: Re: Level3 worldwide emergency upgrade? > OK, having had that first cup of coffee, I can say perhaps the main reason I was wondering is I've gotten used to Level3 always being on top of things (and admittedly, rarely communicating). They've reached the top by often being a black

Re: Level3 worldwide emergency upgrade?

2013-02-06 Thread Justin M. Streiner
On Wed, 6 Feb 2013, Ray Wong wrote: My impression is mostly that people are left feeling uncomfortable by a massive upgrade of this sort with so little communication about why and so on. "Emergency work for five hours and 30 minutes disconnection" that turns out to take longer than 30 minutes o

Re: Level3 worldwide emergency upgrade?

2013-02-06 Thread PC
Given the issue was announced a week ago, I'm surprised they didn't provide some sort of emergency notification prior to the upgrade. However, I certainly understand their immediate desire to deploy this update. I don't think it's bad as the BGP one from not too long ago in that exploit code is n

Re: Level3 worldwide emergency upgrade?

2013-02-06 Thread Ray Wong
> OK, having had that first cup of coffee, I can say perhaps the main reason I was wondering is I've gotten used to Level3 always being on top of things (and admittedly, rarely communicating). They've reached the top by often being a black box of reliability, so it's (perhaps unrealistically) surp

Re: Level3 worldwide emergency upgrade?

2013-02-06 Thread joel jaeggli
On 2/6/13 7:43 AM, Ray Wong wrote: On Wed, Feb 6, 2013 at 7:10 AM, Andrew Sullivan wrote: On Wed, Feb 06, 2013 at 07:39:14AM -0500, Jared Mauch wrote: So, I'm wondering what is shocking that someone may have to push out some sort of upgrade either urgently or periodically that is so impacting

Re: Level3 worldwide emergency upgrade?

2013-02-06 Thread Ray Wong
On Wed, Feb 6, 2013 at 7:10 AM, Andrew Sullivan wrote: > On Wed, Feb 06, 2013 at 07:39:14AM -0500, Jared Mauch wrote: >> >> So, I'm wondering what is shocking that someone may have to push out some >> sort of upgrade either urgently or periodically that is so impacting and >> causes these emails

Re: Level3 worldwide emergency upgrade?

2013-02-06 Thread Andrew Sullivan
On Wed, Feb 06, 2013 at 07:39:14AM -0500, Jared Mauch wrote: > > So, I'm wondering what is shocking that someone may have to push out some > sort of upgrade either urgently or periodically that is so impacting and > causes these emails on the list. > My impression is mostly that people are lef

RE: Level3 worldwide emergency upgrade?

2013-02-06 Thread Alex Rubenstein
> Yeah, perhaps not as elegantly worded as I would have hoped, but there are > many reasons things "go down". Just one of those elements is the internet > part, there's also transport, power, and other elements that combine to > make this complex system called the internet. If you N+N or N+1 your

Re: Level3 worldwide emergency upgrade?

2013-02-06 Thread Jared Mauch
On Feb 6, 2013, at 7:57 AM, Alex Rubenstein wrote: >> Would you rather your ISP not maintain their devices? Are the >> consequences "so bad" of a 30 minute outage that your business >> is severely impacted? >> >> - Jared > > You had me up until that line. > > That should be expanded a littl

Re: Level3 worldwide emergency upgrade?

2013-02-06 Thread Jonathan Towne
On Wed, Feb 06, 2013 at 07:57:06AM -0500, Alex Rubenstein scribbled: # The question should be more along the lines of, "why aren't you multihomed in a way that would make a 30 minute outage (which is inevitable) irrelevant to you? The fun part of this emergency maintenance in the northeast USA w

RE: Level3 worldwide emergency upgrade?

2013-02-06 Thread Alex Rubenstein
> Would you rather your ISP not maintain their devices? Are the > consequences "so bad" of a 30 minute outage that your business > is severely impacted? > > - Jared You had me up until that line. That should be expanded a little ... First, I'd say, yes - many businesses would be severely impa

Re: Level3 worldwide emergency upgrade?

2013-02-06 Thread Jared Mauch
On Feb 6, 2013, at 6:38 AM, Peter Ehiwe wrote: > Also received same ... > > On Wed, Feb 6, 2013 at 10:58 AM, Ray Wong wrote: > >> Does anyone have details on tonight's apparent worldwide emergency >> router upgrade? All I managed to get out of the portal was 30 minutes, >> "Service Affecting"

Re: Level3 worldwide emergency upgrade?

2013-02-06 Thread Peter Ehiwe
Also received same ... On Wed, Feb 6, 2013 at 10:58 AM, Ray Wong wrote: > Does anyone have details on tonight's apparent worldwide emergency > router upgrade? All I managed to get out of the portal was 30 minutes, > "Service Affecting" (no kidding?) and the NOC line gave me the > recording about

Re: Level3 worldwide emergency upgrade?

2013-02-06 Thread Bret Palsson
I just received this email from level3 Summary Level 3 Communications will perform a mandatory network upgrade that will be service impacting and will impact devices in multiple locations. We are upgrading the code on portions of the global network to increase stability for the overall n

Re: Level3 worldwide emergency upgrade?

2013-02-06 Thread Jason Biel
That is general guess. On Wed, Feb 6, 2013 at 5:11 AM, Stephane Bortzmeyer wrote: > On Wed, Feb 06, 2013 at 01:04:40PM +0200, > JP Viljoen wrote > a message of 10 lines which said: > > > the general guess I saw was that it was Juniper-related. > > Juniper Technical Bulletin PSN-2013-01-823, pr

Re: Level3 worldwide emergency upgrade?

2013-02-06 Thread james jones
ugh! On Wed, Feb 6, 2013 at 6:04 AM, JP Viljoen wrote: > On 06 Feb 2013, at 11:58 AM, Ray Wong wrote: > > Does anyone have details on tonight's apparent worldwide emergency > > router upgrade? All I managed to get out of the portal was 30 minutes, > > "Service Affecting" (no kidding?) and the N

Re: Level3 worldwide emergency upgrade?

2013-02-06 Thread Stephane Bortzmeyer
On Wed, Feb 06, 2013 at 01:04:40PM +0200, JP Viljoen wrote a message of 10 lines which said: > the general guess I saw was that it was Juniper-related. Juniper Technical Bulletin PSN-2013-01-823, probably?

Re: Level3 worldwide emergency upgrade?

2013-02-06 Thread JP Viljoen
On 06 Feb 2013, at 11:58 AM, Ray Wong wrote: > Does anyone have details on tonight's apparent worldwide emergency > router upgrade? All I managed to get out of the portal was 30 minutes, > "Service Affecting" (no kidding?) and the NOC line gave me the > recording about it and disconnected me. Not

Level3 worldwide emergency upgrade?

2013-02-06 Thread Ray Wong
Does anyone have details on tonight's apparent worldwide emergency router upgrade? All I managed to get out of the portal was 30 minutes, "Service Affecting" (no kidding?) and the NOC line gave me the recording about it and disconnected me. -R>