Send Outages-discussion mailing list submissions to
outages-discussion@outages.org
To subscribe or unsubscribe via the World Wide Web, visit
https://puck.nether.net/mailman/listinfo/outages-discussion
or, via email, send a message with subject or body 'help' to
outages-discussion-requ...@outages.org
You can reach the person managing the list at
outages-discussion-ow...@outages.org
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Outages-discussion digest..."
Today's Topics:
1. Re: S3 Outages Postmortem (Chapman, Brad (NBCUniversal))
2. Re: [outages] Amazon S3 down (Jay Ashworth)
3. Re: [outages] Amazon S3 down (Jim Popovitch)
----------------------------------------------------------------------
Message: 1
Date: Wed, 1 Mar 2017 19:25:37 +0000
From: "Chapman, Brad (NBCUniversal)" <brad.chap...@nbcuni.com>
To: Kevin Blackham <black...@gmail.com>, Bob Strecansky
<b...@mailchimp.com>
Cc: "outages-discussion@outages.org" <outages-discussion@outages.org>
Subject: Re: [Outages-discussion] S3 Outages Postmortem
Message-ID:
<d8408d7451d5434eb79c289511b88cc410b97...@usushemwp013.mail.tfayd.com>
Content-Type: text/plain; charset="utf-8"
??lots of services affected??
Well, that was pretty obvious from the dashboard yesterday:
https://i.imgur.com/xTec0Bn.png
-Brad
From: Outages-discussion [mailto:outages-discussion-boun...@outages.org] On
Behalf Of Kevin Blackham
Sent: Wednesday, March 1, 2017 11:17 AM
To: Bob Strecansky <b...@mailchimp.com>
Cc: outages-discussion@outages.org
Subject: Re: [Outages-discussion] S3 Outages Postmortem
I have some insights, but I'm under NDA. This was big enough I expect some
public disclosure (my words).
I can tell you we observed lots of services affected, not just S3. EBS was
jacking up IO all over the place, and many machines didn't even ping. SES was
quite broken, as was autoscaling. One might conclude it was a network problem.
On Mar 1, 2017 12:09, "Bob Strecansky"
<b...@mailchimp.com<mailto:b...@mailchimp.com>> wrote:
Has anyone heard anything about why S3 was down for 5 hours yesterday? Usually
Amazon doesn't post postmortems, and i'm curious as to what happened.
Thanks,
Bob Strecansky
--
Thanks,
-B
_______________________________________________
Outages-discussion mailing list
Outages-discussion@outages.org<mailto:Outages-discussion@outages.org>
https://puck.nether.net/mailman/listinfo/outages-discussion
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<https://puck.nether.net/pipermail/outages-discussion/attachments/20170301/51b349c2/attachment-0001.html>
------------------------------
Message: 2
Date: Wed, 01 Mar 2017 19:35:55 -0500
From: Jay Ashworth <j...@baylink.com>
To: outages-discussion@outages.org
Subject: Re: [Outages-discussion] [outages] Amazon S3 down
Message-ID: <37a0ac20-c247-4d30-b353-e026fc703...@baylink.com>
Content-Type: text/plain; charset="utf-8"
The post-mortem on the Oscars is simple, and I had it 30 seconds after the full
explanation of what happened was released:
Each side had a copy of each envelope. Each side *should have had* a trash can,
into which the accountant who did not hand that award's envelope to a presenter
put the other envelope.
They did not.
That is all.
Cheers,
-- jra
On March 1, 2017 1:10:15 PM EST, Frank Bulk <frnk...@iname.com> wrote:
>Here is ThousandEye?s analysis (guess):
>
>https://blog.thousandeyes.com/aws-s3-outage-likely-caused-by-internal-network-issue/
>
>
>
>Frank
>
>
>
>From: Outages [mailto:outages-boun...@outages.org] On Behalf Of
>Chapman, Brad (NBCUniversal) via Outages
>Sent: Wednesday, March 01, 2017 10:44 AM
>To: outa...@outages.org
>Subject: Re: [outages] Amazon S3 down
>
>
>
>Well, that makes *two* post-mortems I look forward to reading about
>this month: the Oscar night best picture kerfuffle, and now this Amazon
>outage.
>
>
>
>
>
>Brad Chapman
>
>Mac Systems Engineer
>
>NBCUniversal
>
>
>
>
>
>From: Outages [mailto:outages-boun...@outages.org] On Behalf Of Jake
>Hayhurst via Outages
>Sent: Tuesday, February 28, 2017 4:12 PM
>To: Ryan Prihoda <rprih...@dynaconnections.com
><mailto:rprih...@dynaconnections.com> >
>Cc: outa...@outages.org <mailto:outa...@outages.org>
>Subject: Re: [outages] Amazon S3 down
>
>
>
>Same here. We were more or less back up about 2 hours ago.
>
>
>
>On 1 Mar 2017 12:10 a.m., "Ryan Prihoda via Outages"
><outa...@outages.org <mailto:outa...@outages.org> > wrote:
>
>We're beginning to see recovery here.
>
>Ryan Prihoda
>Systems Administrator
>
>dynaConnections
>512.306.9898 ext 107 <tel:(512)%20306-9898>
>1101 S. Capital of TX Hwy.
>Bldg. H., Suite 130
>Austin, Texas 78746
>rprih...@dynaconnections.com <mailto:rprih...@dynaconnections.com>
>www.dynaconnections.com <http://www.dynaconnections.com>
>
>On 02/28/2017 11:55 AM, Carlos Alvarez via Outages wrote:
>
>We are unable to reach it from Cox cable, Highwinds, or Cogent from the
>Phoenix area. Also several of the outage testing sites say it's down.
>
>
>
>
>
>
>--
>
>Carlos Alvarez
>
>602-368-6403 <tel:(602)%20368-6403>
>
>
>
>
>
>_______________________________________________
>Outages mailing list
>outa...@outages.org <mailto:outa...@outages.org>
>https://puck.nether.net/mailman/listinfo/outages
>
>
>
>
>_______________________________________________
>Outages mailing list
>outa...@outages.org <mailto:outa...@outages.org>
>https://puck.nether.net/mailman/listinfo/outages
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<https://puck.nether.net/pipermail/outages-discussion/attachments/20170301/5b59cffe/attachment-0001.html>
------------------------------
Message: 3
Date: Wed, 1 Mar 2017 19:45:44 -0500
From: Jim Popovitch <jim...@gmail.com>
To: outages-discussion@outages.org
Subject: Re: [Outages-discussion] [outages] Amazon S3 down
Message-ID:
<CAGfsgR0B0zRZX77=08XpyGef2bg8=+0b9e7+eoxzvsbqrg7...@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8
On Wed, Mar 1, 2017 at 7:35 PM, Jay Ashworth <j...@baylink.com> wrote:
> The post-mortem on the Oscars is simple, and I had it 30 seconds after the
> full explanation of what happened was released:
>
> Each side had a copy of each envelope. Each side *should have had* a trash
> can, into which the accountant who did not hand that award's envelope to a
> presenter put the other envelope.
>
> They did not.
>
> That is all.
Where's the post-mortem on why there were 30+ emails saying there was
an insignificant S3 outage. :-)
-Jim P.
------------------------------
Subject: Digest Footer
_______________________________________________
Outages-discussion mailing list
Outages-discussion@outages.org
https://puck.nether.net/mailman/listinfo/outages-discussion
------------------------------
End of Outages-discussion Digest, Vol 93, Issue 2
*************************************************
