Just wanted to report that we are back up and running for the most
part as well, BUT quite a number of our servers are still experiencing
some BlackOut periods where twitter fails to respond and connections
time out. They seem to last about 5-10 minutes each. We are running
quite a few servers during this time to mitigate the issue propagating
to our users, but it is a costly proposition to continue running this
many servers for extended periods of time.
Here is a traceroute to twitter.com on one of the servers during the
"Blackout" when it is not getting any response from twitter
traceroute to twitter.com (168.143.162.100), 30 hops max, 40 byte
packets
1 67-207-128-2.slicehost.net (67.207.128.2) 0.000 ms 0.000 ms
0.000 ms
2 209-20-79-2.slicehost.net (209.20.79.2) 0.000 ms 0.000 ms
0.000 ms
3 ge-6-10-193.car1.StLouis1.Level3.net (4.53.160.189) 0.000 ms
0.000 ms 0.000 ms
4 ae-11-11.car2.StLouis1.Level3.net (4.69.132.186) 0.000 ms 0.000
ms 0.000 ms
5 ae-4-4.ebr2.Chicago1.Level3.net (4.69.132.190) 7.999 ms 7.999
ms 7.999 ms
6 ae-2-54.edge3.Chicago3.Level3.net (4.68.101.116) 7.999 ms
ae-2-52.edge3.Chicago3.Level3.net (4.68.101.52) 7.999 ms
ae-2-54.edge3.Chicago3.Level3.net (4.68.101.116) 7.999 ms
7 4.68.63.198 (4.68.63.198) 7.999 ms 7.999 ms 8.000 ms
8 ae-1.r21.chcgil09.us.bb.gin.ntt.net (129.250.3.8) 8.000 ms
8.000 ms 8.000 ms
9 as-5.r20.snjsca04.us.bb.gin.ntt.net (129.250.3.77) 51.996 ms
51.996 ms 51.996 ms
10 xe-1-3.r02.mlpsca01.us.bb.gin.ntt.net (129.250.5.61) 55.995 ms
55.995 ms 55.995 ms
11 mg-1.c00.mlpsca01.us.da.verio.net (129.250.24.202) 55.995 ms
55.995 ms 59.995 ms
12 128.121.150.245 (128.121.150.245) 55.995 ms 51.995 ms 51.995 ms
13 128.121.150.245 (128.121.150.245) 51.996 ms !X * *
On Aug 10, 2009, at 2:57 PM, Ryan Sarver wrote:
Wanted to send out a status update and let everyone know where the
situation stands as of today at noon.
- Most developers are reporting being back in operation as of noon
on Sunday
- We have changed our defenses to make sure API developers are
better supported. As such the system has more general strain on it
and thus will produce some more 502/503 errors. If you see them, you
should do a geometric back off instead of just sending a new request.
- OAuth should be fully operational
- If you continue to have unexpected errors, please produce a packet
trace so we can help debug and define the issue.
I will continue to give periodic updates throughout the day as we
know more, but as most apps are back in action the updates will be
more based on new news. Please continue to let us know of any
unexpected issues you may have.
Thanks again for your continued patience and support.
Best, Ryan
