This morning, some tools are not working... I was able to run a console, but not do a backup.
Its not as bad as some services, Quora is still down. On Fri, Apr 22, 2011 at 12:18 AM, Edward Hotchkiss < edw...@groupthinklabs.com> wrote: > What an epic thread. Luckily I'm still a few days away from launching a new > app on Heroku. > > - edward. > > On Thu, Apr 21, 2011 at 1:13 PM, Jeff Schmitz <jeffrey.j.schm...@gmail.com > > wrote: > >> Latest: >> >> I suppose Heroku is in the unavailability zone that is still down. Sorry, >> Freudian slip. >> 12:30 PM PDT We have observed successful new launches of EBS backed >> instances for the past 15 minutes in all but one of the availability zones >> in the US-EAST-1 Region. The team is continuing to work to recover the >> unavailable EBS volumes as quickly as possible. >> >> >> On Thu, Apr 21, 2011 at 1:45 PM, John Norman <j...@7fff.com> wrote: >> >>> Here's what you want -- from: http://status.aws.amazon.com/ >>> >>> The last three provide the most information. >>> >>> 1:41 AM PDT We are currently investigating latency and error rates with >>> EBS volumes and connectivity issues reaching EC2 instances in the US-EAST-1 >>> region. >>> >>> 2:18 AM PDT We can confirm connectivity errors impacting EC2 instances >>> and increased latencies impacting EBS volumes in multiple availability zones >>> in the US-EAST-1 region. Increased error rates are affecting EBS >>> CreateVolume API calls. We continue to work towards resolution. >>> >>> 2:49 AM PDT We are continuing to see connectivity errors impacting EC2 >>> instances, increased latencies impacting EBS volumes in multiple >>> availability zones in the US-EAST-1 region, and increased error rates >>> affecting EBS CreateVolume API calls. We are also experiencing delayed >>> launches for EBS backed EC2 instances in affected availability zones in the >>> US-EAST-1 region. We continue to work towards resolution. >>> >>> 3:20 AM PDT Delayed EC2 instance launches and EBS API error rates are >>> recovering. We're continuing to work towards full resolution. >>> >>> 4:09 AM PDT EBS volume latency and API errors have recovered in one of >>> the two impacted Availability Zones in US-EAST-1. We are continuing to work >>> to resolve the issues in the second impacted Availability Zone. The errors, >>> which started at 12:55AM PDT, began recovering at 2:55am PDT >>> >>> 5:02 AM PDT Latency has recovered for a portion of the impacted EBS >>> volumes. We are continuing to work to resolve the remaining issues with EBS >>> volume latency and error rates in a single Availability Zone. >>> >>> 6:09 AM PDT EBS API errors and volume latencies in the affected >>> availability zone remain. We are continuing to work towards resolution. >>> >>> 6:59 AM PDT There has been a moderate increase in error rates for >>> CreateVolume. This may impact the launch of new EBS-backed EC2 instances in >>> multiple availability zones in the US-EAST-1 region. Launches of instance >>> store AMIs are currently unaffected. We are continuing to work on resolving >>> this issue. >>> >>> 7:40 AM PDT In addition to the EBS volume latencies, EBS-backed >>> instances in the US-EAST-1 region are failing at a high rate. This is due to >>> a high error rate for creating new volumes in this region. >>> >>> 8:54 AM PDT We'd like to provide additional color on what were working >>> on right now (please note that we always know more and understand issues >>> better after we fully recover and dive deep into the post mortem). A >>> networking event early this morning triggered a large amount of re-mirroring >>> of EBS volumes in US-EAST-1. This re-mirroring created a shortage of >>> capacity in one of the US-EAST-1 Availability Zones, which impacted new EBS >>> volume creation as well as the pace with which we could re-mirror and >>> recover affected EBS volumes. Additionally, one of our internal control >>> planes for EBS has become inundated such that it's difficult to create new >>> EBS volumes and EBS backed instances. We are working as quickly as possible >>> to add capacity to that one Availability Zone to speed up the re-mirroring, >>> and working to restore the control plane issue. We're starting to see >>> progress on these efforts, but are not there yet. We will continue to >>> provide updates when we have them. >>> >>> 10:26 AM PDT We have made significant progress in stabilizing the >>> affected EBS control plane service. EC2 API calls that do not involve EBS >>> resources in the affected Availability Zone are now seeing significantly >>> reduced failures and latency and are continuing to recover. We have also >>> brought additional capacity online in the affected Availability Zone and >>> stuck EBS volumes (those that were being remirrored) are beginning to >>> recover. We cannot yet estimate when these volumes will be completely >>> recovered, but we will provide an estimate as soon as we have sufficient >>> data to estimate the recovery. We have all available resources working to >>> restore full service functionality as soon as possible. We will continue to >>> provide updates when we have them. >>> >>> 11:09 AM PDT A number of people have asked us for an ETA on when we'll >>> be fully recovered. We deeply understand why this is important and promise >>> to share this information as soon as we have an estimate that we believe is >>> close to accurate. Our high-level ballpark right now is that the ETA is a >>> few hours. We can assure you that all-hands are on deck to recover as >>> quickly as possible. We will update the community as we have more >>> information. >>> >>> >>> >>> >>> On Thu, Apr 21, 2011 at 1:22 PM, Shannon Perkins < >>> johnshannonperk...@gmail.com> wrote: >>> >>>> I'm a total lurker on this list, but I give a strong second to Eric's >>>> comment. >>>> >>>> Whatever the technical explanation/root-cause turns out to be this is >>>> not acceptable platform behavior. >>>> >>>> Very troubling. >>>> >>>> --sp >>>> >>>> >>>> On Thu, Apr 21, 2011 at 2:06 PM, Eric Anderson >>>> <e...@pixelwareinc.com>wrote: >>>> >>>>> On Apr 21, 11:50 am, Rohit Dewan <dew...@gmail.com> wrote: >>>>> > Does anyone know why Heroku not able to redeploy onto another >>>>> cluster? In >>>>> > general, it would seem prudent to spread applications across the >>>>> various >>>>> > clusters so all apps do not suffer an outage when a single cluster is >>>>> > affected. >>>>> >>>>> I agree completely. I was surprised to see that problems in just one >>>>> of Amazons MANY data centers took Heroku down. Even their own website >>>>> and their own support system are down. I thought the point of the >>>>> cloud is to have your app stay up even if there are problems at one >>>>> data center. >>>>> >>>>> Eric >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Heroku" group. >>>>> To post to this group, send email to heroku@googlegroups.com. >>>>> To unsubscribe from this group, send email to >>>>> heroku+unsubscr...@googlegroups.com. >>>>> For more options, visit this group at >>>>> http://groups.google.com/group/heroku?hl=en. >>>>> >>>>> >>>> >>>> >>>> -- >>>> Shannon Perkins >>>> Editor of Interactive News Technologies >>>> Wired.com >>>> 415-276-4914 >>>> --_--_- >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Heroku" group. >>>> To post to this group, send email to heroku@googlegroups.com. >>>> To unsubscribe from this group, send email to >>>> heroku+unsubscr...@googlegroups.com. >>>> For more options, visit this group at >>>> http://groups.google.com/group/heroku?hl=en. >>>> >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Heroku" group. >>> To post to this group, send email to heroku@googlegroups.com. >>> To unsubscribe from this group, send email to >>> heroku+unsubscr...@googlegroups.com. >>> For more options, visit this group at >>> http://groups.google.com/group/heroku?hl=en. >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Heroku" group. >> To post to this group, send email to heroku@googlegroups.com. >> To unsubscribe from this group, send email to >> heroku+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/heroku?hl=en. >> > > > > -- > Regards, > > ======== > Edward Hotchkiss > http://groupthinklabs.com/ > +1.415.200.7902 > ======== > > -- > You received this message because you are subscribed to the Google Groups > "Heroku" group. > To post to this group, send email to heroku@googlegroups.com. > To unsubscribe from this group, send email to > heroku+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/heroku?hl=en. > -- You received this message because you are subscribed to the Google Groups "Heroku" group. To post to this group, send email to heroku@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
