Re: Facebook Engineering on today's outage

2010-09-24 Thread vijay gill
On Thu, Sep 23, 2010 at 7:17 PM, Jay R. Ashworth wrote: > http://www.facebook.com/notes/facebook-engineering/more-details-on-todays-outage/431441338919 > > Apparently, our surmise about Akamai notwithstanding, the problem was actually > internal to their app-specific caching facilities, which went

Re: Facebook Engineering on today's outage

2010-09-24 Thread Chris Woodfield
Agreed; my reading of this suggests database caching issues (i.e. all the frontend/middleware clients hitting the main sql cluster at once instead of the memcached farm they normally use), not HTTP/CDN caching issues. -C On Sep 23, 2010, at 7:17 12PM, Jay R. Ashworth wrote: > http://www.facebo

Facebook Engineering on today's outage

2010-09-23 Thread Jay R. Ashworth
http://www.facebook.com/notes/facebook-engineering/more-details-on-todays-outage/431441338919 Apparently, our surmise about Akamai notwithstanding, the problem was actually internal to their app-specific caching facilities, which went into Sorcerer's Apprentice mode, and they had to kill them all