Re: FYI Netflix is down

[Grant Ridder]

well one would think that they could at least get power redundancy right...

On Sat, Jun 30, 2012 at 1:07 AM, Roy  wrote:

> On 6/29/2012 10:38 PM, jamie rishaw wrote:
>
>> you know what's happening even more?
>>
>> ..Amazon not learning their lesson.
>>
>> they just had an outage quite similar.. they "performed a full audit" on
>> electrical systems worldwide, according to the rfo/post mortem.
>>
>> looks like they need to perform a "full and we mean it" audit, and like
>> I've been doing/participating in at dot coms for a decade plus: Actually
>> Do
>> Regular Load tests..
>>
>> Related/equally to blame: companies that rely heavily on one aws zone, or
>> arguably "one cloud" (period), are asking for it.
>>
>> Please stop these crappy practices, people.  Do real world DR testing.
>>  Play "What If This City Dropped Off The Map" games, because tonight,
>> parts
>> of VA infact did.
>>
>> ...
>>
>
> I am not a computer science guy but been around a long time.  Data centers
> and clouds are like software.  Once they reach a certain size, its
> impossible to keep the bugs out.  You can test and test your heart out and
> something will slip by.  You can say the same thing about nuclear reactors,
> Apollo moon missions, the NorthEast power grid, and most other technology
> disasters.
>
>
>
>

Re: FYI Netflix is down

[Roy]

On 6/29/2012 10:38 PM, jamie rishaw wrote:

you know what's happening even more?

..Amazon not learning their lesson.

they just had an outage quite similar.. they "performed a full audit" on
electrical systems worldwide, according to the rfo/post mortem.

looks like they need to perform a "full and we mean it" audit, and like
I've been doing/participating in at dot coms for a decade plus: Actually Do
Regular Load tests..

Related/equally to blame: companies that rely heavily on one aws zone, or
arguably "one cloud" (period), are asking for it.

Please stop these crappy practices, people.  Do real world DR testing.
  Play "What If This City Dropped Off The Map" games, because tonight, parts
of VA infact did.

...


I am not a computer science guy but been around a long time.  Data 
centers and clouds are like software.  Once they reach a certain size, 
its impossible to keep the bugs out.  You can test and test your heart 
out and something will slip by.  You can say the same thing about 
nuclear reactors, Apollo moon missions, the NorthEast power grid, and 
most other technology disasters.

Re: FYI Netflix is down

[Bjorn Leffler]

On Sat, Jun 30, 2012 at 3:38 PM, jamie rishaw  wrote:
> ...
> Down: Instagram, Pinterest, Netflix, Heroku, Woot. Pocket(Read It Later),
> and on and on.  A bunch of openID sites.  A bunch of DNS sites (think
> zoneedit et al).  Infact, probably nearly a /12 if not more of space..
> ...

Zoneedit doesn't seem to be down . I can both use the website and
resolve my domains.

Re: FYI Netflix is down

[jamie rishaw]

you know what's happening even more?

..Amazon not learning their lesson.

they just had an outage quite similar.. they "performed a full audit" on
electrical systems worldwide, according to the rfo/post mortem.

looks like they need to perform a "full and we mean it" audit, and like
I've been doing/participating in at dot coms for a decade plus: Actually Do
Regular Load tests..

Related/equally to blame: companies that rely heavily on one aws zone, or
arguably "one cloud" (period), are asking for it.

Please stop these crappy practices, people.  Do real world DR testing.
 Play "What If This City Dropped Off The Map" games, because tonight, parts
of VA infact did.

Down: Instagram, Pinterest, Netflix, Heroku, Woot. Pocket(Read It Later),
and on and on.  A bunch of openID sites.  A bunch of DNS sites (think
zoneedit et al).  Infact, probably nearly a /12 if not more of space..

Blame lies both with AWS (again) and with these services providers.

They all should know better.


-j
On Jun 29, 2012 11:22 PM, "Justin M. Streiner" 
wrote:

> On Fri, 29 Jun 2012, Mike Lyon wrote:
>
>  Whatever happened to UPSs and generators?
>>
>
> They can and do fail.  See list archives for numerous reports and examples
> :)
>
> Generators are capable of not starting.
> ATSs can get into a situation where they don't transfer loads properly, or
> they can't start the generator(s)
> UPSs can fail, drain out, or be left in bypass.
> Breakers can trip and need a manual reset
> etc...
>
> jms
>
>  On Fri, Jun 29, 2012 at 8:45 PM, Jason Baugher > >wrote:
>>
>>  Nature is such a PITA.
>>>
>>>
>>> On 6/29/2012 10:42 PM, James Laszko wrote:
>>>
>>>  To further expand:

 8:21 PM PDT We are investigating connectivity issues for a number of
 instances in the US-EAST-1 Region.

  8:31 PM PDT We are investigating elevated errors rates for APIs in the
 US-EAST-1 (Northern Virginia) region, as well as connectivity issues to
 instances in a single availability zone.

  8:40 PM PDT We can confirm that a large number of instances in a single
 Availability Zone have lost power due to electrical storms in the area.
 We
 are actively working to restore power.

 -Original Message-
 From: Grant Ridder [mailto:shortdudey123@gmail.com<
 shortdudey...@gmail.com>
 ]
 Sent: Friday, June 29, 2012 8:42 PM
 To: Jason Baugher
 Cc: nanog@nanog.org
 Subject: Re: FYI Netflix is down

  From Amazon
>

 Amazon Elastic Compute Cloud (N. Virginia)  (
 http://status.aws.amazon.com/ )
 8:21 PM PDT We are investigating connectivity issues for a number of
 instances in the US-EAST-1 Region.
 8:31 PM PDT We are investigating elevated errors rates for APIs in the
 US-EAST-1 (Northern Virginia) region, as well as connectivity issues to
 instances in a single availability zone.

 -Grant

 On Fri, Jun 29, 2012 at 10:40 PM, Jason Baugher >>>
> wrote:
>

  Seeing some reports of Pinterest and Instagram down as well. Amazon

> cloud services being implicated.
>
>
> On 6/29/2012 10:22 PM, Joe Blanchard wrote:
>
>  Seems that they are unreachable at the moment. Called and theres a
>
>> recorded message stating they are aware of an issue, no details.
>>
>> -Joe
>>
>>
>>
>>
>>
>
>

>>>
>>>
>>>
>>
>> --
>> Mike Lyon
>> 408-621-4826
>> mike.l...@gmail.com
>>
>> http://www.linkedin.com/in/**mlyon 
>>
>>
>

Re: FYI Netflix is down

[Justin M. Streiner]

On Fri, 29 Jun 2012, Mike Lyon wrote:


Whatever happened to UPSs and generators?


They can and do fail.  See list archives for numerous reports and 
examples :)


Generators are capable of not starting.
ATSs can get into a situation where they don't transfer loads properly, or 
they can't start the generator(s)

UPSs can fail, drain out, or be left in bypass.
Breakers can trip and need a manual reset
etc...

jms


On Fri, Jun 29, 2012 at 8:45 PM, Jason Baugher wrote:


Nature is such a PITA.


On 6/29/2012 10:42 PM, James Laszko wrote:


To further expand:

8:21 PM PDT We are investigating connectivity issues for a number of
instances in the US-EAST-1 Region.

 8:31 PM PDT We are investigating elevated errors rates for APIs in the
US-EAST-1 (Northern Virginia) region, as well as connectivity issues to
instances in a single availability zone.

 8:40 PM PDT We can confirm that a large number of instances in a single
Availability Zone have lost power due to electrical storms in the area. We
are actively working to restore power.

-Original Message-
From: Grant Ridder [mailto:shortdudey123@gmail.**com
]
Sent: Friday, June 29, 2012 8:42 PM
To: Jason Baugher
Cc: nanog@nanog.org
Subject: Re: FYI Netflix is down


From Amazon


Amazon Elastic Compute Cloud (N. Virginia)  (
http://status.aws.amazon.com/**)
8:21 PM PDT We are investigating connectivity issues for a number of
instances in the US-EAST-1 Region.
8:31 PM PDT We are investigating elevated errors rates for APIs in the
US-EAST-1 (Northern Virginia) region, as well as connectivity issues to
instances in a single availability zone.

-Grant

On Fri, Jun 29, 2012 at 10:40 PM, Jason Baugher 
wrote:


 Seeing some reports of Pinterest and Instagram down as well. Amazon

cloud services being implicated.


On 6/29/2012 10:22 PM, Joe Blanchard wrote:

 Seems that they are unreachable at the moment. Called and theres a

recorded message stating they are aware of an issue, no details.

-Joe
















--
Mike Lyon
408-621-4826
mike.l...@gmail.com

http://www.linkedin.com/in/mlyon

Re: FYI Netflix is down

[Seth Mattinen]

On 6/29/12 8:22 PM, Joe Blanchard wrote:
> Seems that they are unreachable at the moment. Called and theres a recorded
> message stating they are aware of an issue, no details.
> 


Streaming services and web; just tried my Roku and it failed to connect.

~Seth

Re: FYI Netflix is down

[William Herrin]

On Fri, Jun 29, 2012 at 11:42 PM, Grant Ridder  wrote:
> From Amazon
>
> Amazon Elastic Compute Cloud (N. Virginia)  (http://status.aws.amazon.com/)
> 8:21 PM PDT We are investigating connectivity issues for a number of
> instances in the US-EAST-1 Region.
> 8:31 PM PDT We are investigating elevated errors rates for APIs in the
> US-EAST-1 (Northern Virginia) region, as well as connectivity issues to
> instances in a single availability zone.

Major storm:

http://www.washingtonpost.com/blogs/capital-weather-gang/post/severe-thunderstorm-watch-through-1-am-for-washington-dc-area/2012/06/29/gJQAY04LCW_blog.html

"Storms packing wind gusts of nearly 80 mph have just blown through
the D.C.-Baltimore region"

https://www.dom.com/storm-center/dominion-electric-outage-summary.jsp

Right around 50% of northern Virginia is without power right now.

Regards,
Bill Herrin

Whose generator worked although the first five gas stations I passed
had no power.




-- 
William D. Herrin  her...@dirtside.com  b...@herrin.us
3005 Crane Dr. .. Web: 
Falls Church, VA 22042-3004

Re: FYI Netflix is down

[Grant Ridder]

8:49 PM PDT Power has been restored to the impacted Availability Zone and
we are working to bring impacted instances and volumes back online

On Fri, Jun 29, 2012 at 10:52 PM, Grant Ridder wrote:

> They may use it for content, but reddit.com resolves to IPs own by quest
>
>
> On Fri, Jun 29, 2012 at 10:51 PM, Seth Mattinen wrote:
>
>> On 6/29/12 8:47 PM, Mike Lyon wrote:
>> > Whatever happened to UPSs and generators?
>> >
>>
>> You don't need them with The Cloud!
>>
>> But seriously, this is something like the third or fourth time AWS fell
>> over flat in recent memory.
>>
>> ~Seth
>>
>>
>>
>>
>

Re: FYI Netflix is down

[Grant Ridder]

They may use it for content, but reddit.com resolves to IPs own by quest

On Fri, Jun 29, 2012 at 10:51 PM, Seth Mattinen  wrote:

> On 6/29/12 8:47 PM, Mike Lyon wrote:
> > Whatever happened to UPSs and generators?
> >
>
> You don't need them with The Cloud!
>
> But seriously, this is something like the third or fourth time AWS fell
> over flat in recent memory.
>
> ~Seth
>
>
>
>

Re: FYI Netflix is down

[Seth Mattinen]

On 6/29/12 8:47 PM, Mike Lyon wrote:
> Whatever happened to UPSs and generators?
> 

You don't need them with The Cloud!

But seriously, this is something like the third or fourth time AWS fell
over flat in recent memory.

~Seth

Re: FYI Netflix is down

[Derek Ivey]

I was wondering the same thing! Also, Reddit appears to be really slow 
right now and I keep getting "reddit is under heavy load right now, 
sorry. Try again in a few minutes."


I wonder if it's related. I believe they use Amazon for some of their stuff.

Derek

On 6/29/2012 11:47 PM, Mike Lyon wrote:

Whatever happened to UPSs and generators?

On Fri, Jun 29, 2012 at 8:45 PM, Jason Baugher wrote:


Nature is such a PITA.


On 6/29/2012 10:42 PM, James Laszko wrote:


To further expand:

8:21 PM PDT We are investigating connectivity issues for a number of
instances in the US-EAST-1 Region.

  8:31 PM PDT We are investigating elevated errors rates for APIs in the
US-EAST-1 (Northern Virginia) region, as well as connectivity issues to
instances in a single availability zone.

  8:40 PM PDT We can confirm that a large number of instances in a single
Availability Zone have lost power due to electrical storms in the area. We
are actively working to restore power.

-Original Message-
From: Grant Ridder [mailto:shortdudey123@gmail.**com
]
Sent: Friday, June 29, 2012 8:42 PM
To: Jason Baugher
Cc: nanog@nanog.org
Subject: Re: FYI Netflix is down

>From Amazon

Amazon Elastic Compute Cloud (N. Virginia)  (
http://status.aws.amazon.com/**)
8:21 PM PDT We are investigating connectivity issues for a number of
instances in the US-EAST-1 Region.
8:31 PM PDT We are investigating elevated errors rates for APIs in the
US-EAST-1 (Northern Virginia) region, as well as connectivity issues to
instances in a single availability zone.

-Grant

On Fri, Jun 29, 2012 at 10:40 PM, Jason Baugher 
wrote:

  Seeing some reports of Pinterest and Instagram down as well. Amazon

cloud services being implicated.


On 6/29/2012 10:22 PM, Joe Blanchard wrote:

  Seems that they are unreachable at the moment. Called and theres a

recorded message stating they are aware of an issue, no details.

-Joe

Re: FYI Netflix is down

[Grant Ridder]

Yes, although, when you launch an instance, you do have the option of
selecting a zone if you want.  However, once the instance is started it
stays in that zone and does not switch.

On Fri, Jun 29, 2012 at 10:47 PM, Ian Wilson  wrote:

> On Fri, Jun 29, 2012 at 11:44 PM, Grant Ridder 
> wrote:
> > I have an instance in zone C and it is up and fine, so it must be A, B,
> or
> > D that is down.
>
> It is my understanding that instance zones are randomized between
> customers -- so your zone C may be my zone A.
>
> Ian
> --
> Ian Wilson
> ian.m.wil...@gmail.com
>
> Solving site load issues with database replication is a lot like
> solving your own personal problems with heroin -- at first, it sorta
> works, but after a while things just get out of hand.
>

Re: FYI Netflix is down

[Ian Wilson]

On Fri, Jun 29, 2012 at 11:44 PM, Grant Ridder  wrote:
> I have an instance in zone C and it is up and fine, so it must be A, B, or
> D that is down.

It is my understanding that instance zones are randomized between
customers -- so your zone C may be my zone A.

Ian
-- 
Ian Wilson
ian.m.wil...@gmail.com

Solving site load issues with database replication is a lot like
solving your own personal problems with heroin -- at first, it sorta
works, but after a while things just get out of hand.

Re: FYI Netflix is down

[Mike Lyon]

Whatever happened to UPSs and generators?

On Fri, Jun 29, 2012 at 8:45 PM, Jason Baugher wrote:

> Nature is such a PITA.
>
>
> On 6/29/2012 10:42 PM, James Laszko wrote:
>
>> To further expand:
>>
>> 8:21 PM PDT We are investigating connectivity issues for a number of
>> instances in the US-EAST-1 Region.
>>
>>  8:31 PM PDT We are investigating elevated errors rates for APIs in the
>> US-EAST-1 (Northern Virginia) region, as well as connectivity issues to
>> instances in a single availability zone.
>>
>>  8:40 PM PDT We can confirm that a large number of instances in a single
>> Availability Zone have lost power due to electrical storms in the area. We
>> are actively working to restore power.
>>
>> -Original Message-
>> From: Grant Ridder [mailto:shortdudey123@gmail.**com
>> ]
>> Sent: Friday, June 29, 2012 8:42 PM
>> To: Jason Baugher
>> Cc: nanog@nanog.org
>> Subject: Re: FYI Netflix is down
>>
>> >From Amazon
>>
>> Amazon Elastic Compute Cloud (N. Virginia)  (
>> http://status.aws.amazon.com/**)
>> 8:21 PM PDT We are investigating connectivity issues for a number of
>> instances in the US-EAST-1 Region.
>> 8:31 PM PDT We are investigating elevated errors rates for APIs in the
>> US-EAST-1 (Northern Virginia) region, as well as connectivity issues to
>> instances in a single availability zone.
>>
>> -Grant
>>
>> On Fri, Jun 29, 2012 at 10:40 PM, Jason Baugher > >wrote:
>>
>>  Seeing some reports of Pinterest and Instagram down as well. Amazon
>>> cloud services being implicated.
>>>
>>>
>>> On 6/29/2012 10:22 PM, Joe Blanchard wrote:
>>>
>>>  Seems that they are unreachable at the moment. Called and theres a
 recorded message stating they are aware of an issue, no details.

 -Joe




>>>
>>>
>>
>
>
>


-- 
Mike Lyon
408-621-4826
mike.l...@gmail.com

http://www.linkedin.com/in/mlyon

Re: FYI Netflix is down

[Jason Baugher]

Nature is such a PITA.

On 6/29/2012 10:42 PM, James Laszko wrote:

To further expand:

8:21 PM PDT We are investigating connectivity issues for a number of instances 
in the US-EAST-1 Region.

  8:31 PM PDT We are investigating elevated errors rates for APIs in the 
US-EAST-1 (Northern Virginia) region, as well as connectivity issues to 
instances in a single availability zone.

  8:40 PM PDT We can confirm that a large number of instances in a single 
Availability Zone have lost power due to electrical storms in the area. We are 
actively working to restore power.

-Original Message-
From: Grant Ridder [mailto:shortdudey...@gmail.com]
Sent: Friday, June 29, 2012 8:42 PM
To: Jason Baugher
Cc: nanog@nanog.org
Subject: Re: FYI Netflix is down

>From Amazon

Amazon Elastic Compute Cloud (N. Virginia)  (http://status.aws.amazon.com/)
8:21 PM PDT We are investigating connectivity issues for a number of instances 
in the US-EAST-1 Region.
8:31 PM PDT We are investigating elevated errors rates for APIs in the
US-EAST-1 (Northern Virginia) region, as well as connectivity issues to 
instances in a single availability zone.

-Grant

On Fri, Jun 29, 2012 at 10:40 PM, Jason Baugher wrote:


Seeing some reports of Pinterest and Instagram down as well. Amazon
cloud services being implicated.


On 6/29/2012 10:22 PM, Joe Blanchard wrote:


Seems that they are unreachable at the moment. Called and theres a
recorded message stating they are aware of an issue, no details.

-Joe

Re: FYI Netflix is down

[Grant Ridder]

I have an instance in zone C and it is up and fine, so it must be A, B, or
D that is down.

On Fri, Jun 29, 2012 at 10:42 PM, James Laszko wrote:

> To further expand:
>
> 8:21 PM PDT We are investigating connectivity issues for a number of
> instances in the US-EAST-1 Region.
>
>  8:31 PM PDT We are investigating elevated errors rates for APIs in the
> US-EAST-1 (Northern Virginia) region, as well as connectivity issues to
> instances in a single availability zone.
>
>  8:40 PM PDT We can confirm that a large number of instances in a single
> Availability Zone have lost power due to electrical storms in the area. We
> are actively working to restore power.
>
> -Original Message-
> From: Grant Ridder [mailto:shortdudey...@gmail.com]
> Sent: Friday, June 29, 2012 8:42 PM
> To: Jason Baugher
> Cc: nanog@nanog.org
> Subject: Re: FYI Netflix is down
>
> From Amazon
>
> Amazon Elastic Compute Cloud (N. Virginia)  (http://status.aws.amazon.com/
> )
> 8:21 PM PDT We are investigating connectivity issues for a number of
> instances in the US-EAST-1 Region.
> 8:31 PM PDT We are investigating elevated errors rates for APIs in the
> US-EAST-1 (Northern Virginia) region, as well as connectivity issues to
> instances in a single availability zone.
>
> -Grant
>
> On Fri, Jun 29, 2012 at 10:40 PM, Jason Baugher  >wrote:
>
> > Seeing some reports of Pinterest and Instagram down as well. Amazon
> > cloud services being implicated.
> >
> >
> > On 6/29/2012 10:22 PM, Joe Blanchard wrote:
> >
> >> Seems that they are unreachable at the moment. Called and theres a
> >> recorded message stating they are aware of an issue, no details.
> >>
> >> -Joe
> >>
> >>
> >>
> >
> >
> >
>

RE: FYI Netflix is down

[James Laszko]

To further expand:

8:21 PM PDT We are investigating connectivity issues for a number of instances 
in the US-EAST-1 Region.

 8:31 PM PDT We are investigating elevated errors rates for APIs in the 
US-EAST-1 (Northern Virginia) region, as well as connectivity issues to 
instances in a single availability zone.

 8:40 PM PDT We can confirm that a large number of instances in a single 
Availability Zone have lost power due to electrical storms in the area. We are 
actively working to restore power.

-Original Message-
From: Grant Ridder [mailto:shortdudey...@gmail.com] 
Sent: Friday, June 29, 2012 8:42 PM
To: Jason Baugher
Cc: nanog@nanog.org
Subject: Re: FYI Netflix is down

>From Amazon

Amazon Elastic Compute Cloud (N. Virginia)  (http://status.aws.amazon.com/)
8:21 PM PDT We are investigating connectivity issues for a number of instances 
in the US-EAST-1 Region.
8:31 PM PDT We are investigating elevated errors rates for APIs in the
US-EAST-1 (Northern Virginia) region, as well as connectivity issues to 
instances in a single availability zone.

-Grant

On Fri, Jun 29, 2012 at 10:40 PM, Jason Baugher wrote:

> Seeing some reports of Pinterest and Instagram down as well. Amazon 
> cloud services being implicated.
>
>
> On 6/29/2012 10:22 PM, Joe Blanchard wrote:
>
>> Seems that they are unreachable at the moment. Called and theres a 
>> recorded message stating they are aware of an issue, no details.
>>
>> -Joe
>>
>>
>>
>
>
>

Re: FYI Netflix is down

[Grant Ridder]

>From Amazon

Amazon Elastic Compute Cloud (N. Virginia)  (http://status.aws.amazon.com/)
8:21 PM PDT We are investigating connectivity issues for a number of
instances in the US-EAST-1 Region.
8:31 PM PDT We are investigating elevated errors rates for APIs in the
US-EAST-1 (Northern Virginia) region, as well as connectivity issues to
instances in a single availability zone.

-Grant

On Fri, Jun 29, 2012 at 10:40 PM, Jason Baugher wrote:

> Seeing some reports of Pinterest and Instagram down as well. Amazon cloud
> services being implicated.
>
>
> On 6/29/2012 10:22 PM, Joe Blanchard wrote:
>
>> Seems that they are unreachable at the moment. Called and theres a
>> recorded
>> message stating they are aware of an issue, no details.
>>
>> -Joe
>>
>>
>>
>
>
>

Re: FYI Netflix is down

[Jason Baugher]

Seeing some reports of Pinterest and Instagram down as well. Amazon 
cloud services being implicated.


On 6/29/2012 10:22 PM, Joe Blanchard wrote:

Seems that they are unreachable at the moment. Called and theres a recorded
message stating they are aware of an issue, no details.

-Joe

FYI Netflix is down

[Joe Blanchard]

Seems that they are unreachable at the moment. Called and theres a recorded
message stating they are aware of an issue, no details.

-Joe

Re: http/ssl to dropbox.com dying

[Israel G. Lugo]

Hi,

On 06/29/2012 11:20 PM, Greg Ihnen wrote:
> It seems like a transport issue.
>
> Is there any tools for checking where an https connection is failing, like a 
> traceroute for https?

GNU/Linux traceroute sends UDP by default. Something along the way could
be filtering UDP, so default traceroute may not be indicative.

To better replicate the problem, you can tell traceroute to send TCP
SYNs to the specific port you're trying to reach (443). Run this as root
(it needs raw sockets):

# traceroute -M tcp -p 443 dropbox.com

Regards,
Israel G. Lugo

http/ssl to dropbox.com dying

[Greg Ihnen]

From other geographic locations I can connect to the dropbox service and get to 
their https web page, but from my home connection I can't, unless I vpn around 
the issue.

downforeveryoneorjustme says it's just me, but they're located someplace else 
geographically, and I don't know if they check the https site. 
http://www.dropbox.com immediately redirects to https://www.dropbox.com

It seems like a transport issue.

Is there any tools for checking where an https connection is failing, like a 
traceroute for https?

I'm not sure if the traceroute results are indicative but here it is

Macintosh-2:~ gregihnen$ traceroute dropbox.com
traceroute: Warning: dropbox.com has multiple addresses; using 199.47.216.179
traceroute to dropbox.com (199.47.216.179), 64 hops max, 52 byte packets
 1  router (192.168.7.1)  1786.458 ms  1.670 ms  2.072 ms
 2  modem (100.42.12.241)  1644.717 ms  2031.032 ms  2113.805 ms
 3  75.7.64.12 (75.7.64.12)  2594.284 ms  1650.347 ms  822.159 ms
 4  75.7.64.2 (75.7.64.2)  1528.550 ms  2168.641 ms  1922.285 ms
 5  12.91.131.205 (12.91.131.205)  2323.903 ms  3137.965 ms  2138.427 ms
 6  cr83.cgcil.ip.att.net (12.122.133.202)  1629.569 ms  1946.842 ms  1621.351 
ms
 7  cr1.cgcil.ip.att.net (12.123.7.110)  2256.595 ms  1515.060 ms  2418.845 ms
 8  gar8.cgcil.ip.att.net (12.122.133.161)  2349.706 ms  2339.392 ms  583.224 ms
 9  192.205.37.150 (192.205.37.150)  1396.288 ms  1732.779 ms  2664.270 ms
10  4.69.158.138 (4.69.158.138)  2690.646 ms
4.69.158.130 (4.69.158.130)  2313.195 ms
4.69.158.138 (4.69.158.138)  1261.560 ms
11  ae-3-3.ebr2.denver1.level3.net (4.69.132.61)  1476.892 ms  1819.138 ms  
2188.664 ms
12  ae-1-100.ebr1.denver1.level3.net (4.69.151.181)  1490.142 ms  2916.895 ms  
2569.848 ms
13  ae-3-3.ebr2.sanjose1.level3.net (4.69.132.57)  4328.125 ms  3226.550 ms  
2648.859 ms
14  ae-72-72.csw2.sanjose1.level3.net (4.69.153.22)  2171.863 ms
ae-82-82.csw3.sanjose1.level3.net (4.69.153.26)  2675.059 ms
ae-92-92.csw4.sanjose1.level3.net (4.69.153.30)  4404.724 ms
15  ae-1-60.edge2.sanjose3.level3.net (4.69.152.17)  3331.595 ms
ae-2-70.edge2.sanjose3.level3.net (4.69.152.81)  3112.938 ms  2492.688 ms
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *
31  * * *
32  * * *
33  * * *
34  * * *
35  * * *
36  * * *

Greg

The Cidr Report

[cidr-report]

This report has been generated at Fri Jun 29 21:12:59 2012 AEST.
The report analyses the BGP Routing Table of AS2.0 router
and generates a report on aggregation potential within the table.

Check http://www.cidr-report.org for a current version of this report.

Recent Table History
Date  PrefixesCIDR Agg
22-06-12417401  241597
23-06-12417626  242305
24-06-12417693  242428
25-06-12417814  241881
26-06-12417414  242356
27-06-12417776  242461
28-06-12417970  242449
29-06-12417364  242229


AS Summary
 41522  Number of ASes in routing system
 17336  Number of ASes announcing only one prefix
  3397  Largest number of prefixes announced by an AS
AS6389 : BELLSOUTH-NET-BLK - BellSouth.net Inc.
  113115104  Largest address span announced by an AS (/32s)
AS4134 : CHINANET-BACKBONE No.31,Jin-rong Street


Aggregation Summary
The algorithm used in this report proposes aggregation only
when there is a precise match using the AS path, so as 
to preserve traffic transit policies. Aggregation is also
proposed across non-advertised address space ('holes').

 --- 29Jun12 ---
ASnumNetsNow NetsAggr  NetGain   % Gain   Description

Table 417485   242253   17523242.0%   All ASes

AS6389  3397  192 320594.3%   BELLSOUTH-NET-BLK -
   BellSouth.net Inc.
AS7029  3246 1630 161649.8%   WINDSTREAM - Windstream
   Communications Inc
AS22773 1655  136 151991.8%   ASN-CXA-ALL-CCI-22773-RDC -
   Cox Communications Inc.
AS4766  2715 1299 141652.2%   KIXS-AS-KR Korea Telecom
AS28573 1967  554 141371.8%   NET Servicos de Comunicao S.A.
AS18566 2090  705 138566.3%   COVAD - Covad Communications
   Co.
AS2118  1288   14 127498.9%   RELCOM-AS OOO "NPO Relcom"
AS4323  1575  386 118975.5%   TWTC - tw telecom holdings,
   inc.
AS10620 1980  803 117759.4%   Telmex Colombia S.A.
AS1785  1930  813 111757.9%   AS-PAETEC-NET - PaeTec
   Communications, Inc.
AS4755  1597  542 105566.1%   TATACOMM-AS TATA
   Communications formerly VSNL
   is Leading ISP
AS7303  1455  460  99568.4%   Telecom Argentina S.A.
AS17974 1971 1091  88044.6%   TELKOMNET-AS2-AP PT
   Telekomunikasi Indonesia
AS7552  1100  221  87979.9%   VIETEL-AS-AP Vietel
   Corporation
AS8151  1375  563  81259.1%   Uninet S.A. de C.V.
AS18101  947  159  78883.2%   RELIANCE-COMMUNICATIONS-IN
   Reliance Communications
   Ltd.DAKC MUMBAI
AS4808  1109  347  76268.7%   CHINA169-BJ CNCGROUP IP
   network China169 Beijing
   Province Network
AS9394   892  162  73081.8%   CRNET CHINA RAILWAY
   Internet(CRNET)
AS13977  839  123  71685.3%   CTELCO - FAIRPOINT
   COMMUNICATIONS, INC.
AS3356    467  64458.0%   LEVEL3 Level 3 Communications
AS855688   57  63191.7%   CANET-ASN-4 - Bell Aliant
   Regional Communications, Inc.
AS30036 1446  825  62142.9%   MEDIACOM-ENTERPRISE-BUSINESS -
   Mediacom Communications Corp
AS17676  692   75  61789.2%   GIGAINFRA Softbank BB Corp.
AS4780   842  246  59670.8%   SEEDNET Digital United Inc.
AS22561 1024  428  59658.2%   DIGITAL-TELEPORT - Digital
   Teleport Inc.
AS19262  998  405  59359.4%   VZGNI-TRANSIT - Verizon Online
   LLC
AS24560 1035  461  57455.5%   AIRTELBROADBAND-AS-AP Bharti
   Airtel Ltd., Telemedia
   Services
AS8452  1270  713  55743.9%   TE-AS TE-AS
AS4804   649   97  55285.1%   MPX-AS Microplex PTY LTD
AS22047  583   31  55294.7%   VTR BANDA ANCHA S.A.

Total  43466140052946167.8%   Top 30 to

BGP Update Report

[cidr-report]

BGP Update Report
Interval: 21-Jun-12 -to- 28-Jun-12 (7 days)
Observation Point: BGP Peering with AS131072

TOP 20 Unstable Origin AS
Rank ASNUpds %  Upds/PfxAS-Name
 1 - AS840255563  1.9%  28.1 -- CORBINA-AS OJSC "Vimpelcom"
 2 - AS982941479  1.5%  31.8 -- BSNL-NIB National Internet 
Backbone
 3 - AS13188   38016  1.3%  70.5 -- BANKINFORM-AS TOV "Bank-Inform"
 4 - AS580035008  1.2% 134.6 -- DNIC-ASBLK-05800-06055 - DoD 
Network Information Center
 5 - AS24560   30326  1.1%  29.3 -- AIRTELBROADBAND-AS-AP Bharti 
Airtel Ltd., Telemedia Services
 6 - AS12479   29579  1.0%  38.0 -- UNI2-AS France Telecom Espana SA
 7 - AS845225140  0.9%  19.1 -- TE-AS TE-AS
 8 - AS28573   25016  0.9%  12.6 -- NET Servicos de Comunicao S.A.
 9 - AS17813   21150  0.7% 159.0 -- MTNL-AP Mahanagar Telephone 
Nigam Ltd.
10 - AS755220962  0.7%  19.0 -- VIETEL-AS-AP Vietel Corporation
11 - AS269720353  0.7% 100.8 -- ERX-ERNET-AS Education and 
Research Network
12 - AS13118   19764  0.7% 411.8 -- ASN-YARTELECOM OJSC Rostelecom
13 - AS20115   18732  0.7%  11.3 -- CHARTER-NET-HKY-NC - Charter 
Communications
14 - AS17621   17293  0.6% 113.8 -- CNCGROUP-SH China Unicom 
Shanghai network
15 - AS24863   16472  0.6%  18.9 -- LINKdotNET-AS
16 - AS19361   15264  0.5% 448.9 -- Atrium Telecomunicacoes Ltda
17 - AS702913362  0.5%   3.9 -- WINDSTREAM - Windstream 
Communications Inc
18 - AS432312157  0.4%   7.7 -- TWTC - tw telecom holdings, inc.
19 - AS26615   11933  0.4%  13.0 -- Tim Celular S.A.
20 - AS34875   10844  0.4%  62.7 -- YANFES OJSC "Uralsviazinform"


TOP 20 Unstable Origin AS (Updates per announced prefix)
Rank ASNUpds %  Upds/PfxAS-Name
 1 - AS408482543  0.1%2543.0 -- FMFCU - Franklin Mint FCU
 2 - AS31893  0.1%1960.0 -- OOO-HAS-TELEKOM-AS OOO 
HAS-TELEKOM
 3 - AS31409  0.1%2120.0 -- OOO-HAS-TELEKOM-AS OOO 
HAS-TELEKOM
 4 - AS29126 932  0.0% 932.0 -- DATIQ-AS Datiq B.V.
 5 - AS55665 835  0.0% 835.0 -- STMI-AS-ID PT Sampoerna 
Telemedia Indonesia
 6 - AS45286 824  0.0% 824.0 -- EDIINDONESIA-AS-ID PT EDI 
INDONESIA
 7 - AS30944 733  0.0% 733.0 -- DKD-AS Bendra Lietuvos, JAV ir 
Rusijos imone uzdaroji akcine bendrove "DKD"
 8 - AS196752229  0.1% 557.2 -- VCS-AS - Virtacore Systems Inc
 9 - AS381428093  0.3% 505.8 -- UNAIR-AS-ID Universitas 
Airlangga
10 - AS19361   15264  0.5% 448.9 -- Atrium Telecomunicacoes Ltda
11 - AS57105 854  0.0% 427.0 -- SUNIMPROF-AS Sunimprof Import 
Export SRL
12 - AS13118   19764  0.7% 411.8 -- ASN-YARTELECOM OJSC Rostelecom
13 - AS27771 812  0.0% 406.0 -- Instituto Venezolano de 
Investigaciones Cientificas
14 - AS194064345  0.1% 395.0 -- TWRS-MA - Towerstream I, Inc.
15 - AS165351122  0.0% 374.0 -- ECHOS-3 - Echostar Holding 
Purchasing Corporation
16 - AS57201 367  0.0% 367.0 -- EDF-AS Estonian Defence Forces
17 - AS48068 709  0.0% 354.5 -- VISONIC Visonic Ltd
18 - AS23533 338  0.0% 338.0 -- PIP-ASN - Permian Investment 
Partners, LP
19 - AS324441805  0.1% 300.8 -- SAFELINK-JEROME1ASN - Safelink 
Internet
20 - AS37429 872  0.0% 290.7 -- Spidernet


TOP 20 Unstable Prefixes
Rank Prefix Upds % Origin AS -- AS Name
 1 - 109.161.64.0/19   19381  0.7%   AS13118 -- ASN-YARTELECOM OJSC Rostelecom
 2 - 220.196.26.0/24   16991  0.6%   AS17621 -- CNCGROUP-SH China Unicom 
Shanghai network
 3 - 159.224.176.0/22  12498  0.4%   AS13188 -- BANKINFORM-AS TOV "Bank-Inform"
 4 - 159.224.222.0/23  12496  0.4%   AS13188 -- BANKINFORM-AS TOV "Bank-Inform"
 5 - 41.43.147.0/2411741  0.4%   AS8452  -- TE-AS TE-AS
 6 - 182.64.0.0/16 10758  0.4%   AS24560 -- AIRTELBROADBAND-AS-AP Bharti 
Airtel Ltd., Telemedia Services
 7 - 122.161.0.0/16 8770  0.3%   AS24560 -- AIRTELBROADBAND-AS-AP Bharti 
Airtel Ltd., Telemedia Services
 8 - 62.36.252.0/22 8135  0.3%   AS12479 -- UNI2-AS France Telecom Espana SA
 9 - 62.36.249.0/24 6626  0.2%   AS12479 -- UNI2-AS France Telecom Espana SA
10 - 202.56.215.0/246579  0.2%   AS24560 -- AIRTELBROADBAND-AS-AP Bharti 
Airtel Ltd., Telemedia Services
11 - 62.36.241.0/24 6245  0.2%   AS12479 -- UNI2-AS France Telecom Espana SA
12 - 62.36.210.0/24 6099  0.2%   AS12479 -- UNI2-AS France Telecom Espana SA
13 - 59.177.48.0/20 5954  0.2%   AS17813 -- MTNL-AP Mahanagar Telephone 
Nigam Ltd.
14 - 194.63.9.0/24  5160  0.2%   AS1273  -- CW Cable and Wireless Worldwide 
plc
15 - 69.38.178.0/24 4310  0.1%   AS19406 -- TWRS-MA

Re: technical contact at ATT Wireless

[Owen DeLong]

> Let it be known that I hate NAT with the burning passion of a million
> suns. But I'm the junior in my workplace, and this is the advice of
> the head honchos. I can easily see both sides of this. I would say
> with a few implementations, (maybe 25 or fewer) NATing isn't that
> difficult.
> 
> Granted we both know that NAT breaks basically everything and makes
> troubleshooting a TON MORE FUN. But plenty of people out there (my
> workplace included) would argue this till the cows come home.

Yep... While this environment would benefit greatly from deploying IPv6 on both 
sides of the connection, the reality is that NAT is easy enough and works well 
enough for the implementor that they will leave it's various pain points for 
the people that have to deal with it after implementation and they won't select 
IPv6 as a solution because it would involve slightly more pain up front.

However, the networks on both sides of these equations will have to face IPv6 
in the relatively near future anyway, unless they aren't actually talking to 
the internet in which case, it doesn't really matter what addresses or 
protocols they use.

Owen

Weekly Routing Table Report

[Routing Analysis Role Account]

This is an automated weekly mailing describing the state of the Internet
Routing Table as seen from APNIC's router in Japan.

The posting is sent to APOPS, NANOG, AfNOG, AusNOG, SANOG, PacNOG, LacNOG,
TRNOG, CaribNOG and the RIPE Routing Working Group.

Daily listings are sent to bgp-st...@lists.apnic.net

For historical data, please see http://thyme.rand.apnic.net.

If you have any comments please contact Philip Smith .

Routing Table Report   04:00 +10GMT Sat 30 Jun, 2012

Report Website: http://thyme.rand.apnic.net
Detailed Analysis:  http://thyme.rand.apnic.net/current/

Analysis Summary


BGP routing table entries examined:  414947
Prefixes after maximum aggregation:  175422
Deaggregation factor:  2.37
Unique aggregates announced to Internet: 202271
Total ASes present in the Internet Routing Table: 41386
Prefixes per ASN: 10.03
Origin-only ASes present in the Internet Routing Table:   33302
Origin ASes announcing only one prefix:   15652
Transit ASes present in the Internet Routing Table:5545
Transit-only ASes present in the Internet Routing Table:133
Average AS path length visible in the Internet Routing Table:   4.5
Max AS path length visible:  27
Max AS path prepend of ASN ( 51742)  24
Prefixes from unregistered ASNs in the Routing Table:   388
Unregistered ASNs in the Routing Table: 122
Number of 32-bit ASNs allocated by the RIRs:   2907
Number of 32-bit ASNs visible in the Routing Table:2539
Prefixes from 32-bit ASNs in the Routing Table:6532
Special use prefixes present in the Routing Table:2
Prefixes being announced from unallocated address space:184
Number of addresses announced to Internet:   2570382252
Equivalent to 153 /8s, 52 /16s and 235 /24s
Percentage of available address space announced:   69.3
Percentage of allocated address space announced:   69.4
Percentage of available address space allocated:   99.9
Percentage of address space in use by end-sites:   93.0
Total number of prefixes smaller than registry allocations:  143966

APNIC Region Analysis Summary
-

Prefixes being announced by APNIC Region ASes:   101216
Total APNIC prefixes after maximum aggregation:   32691
APNIC Deaggregation factor:3.10
Prefixes being announced from the APNIC address blocks:  101660
Unique aggregates announced from the APNIC address blocks:41966
APNIC Region origin ASes present in the Internet Routing Table:4712
APNIC Prefixes per ASN:   21.57
APNIC Region origin ASes announcing only one prefix:   1241
APNIC Region transit ASes present in the Internet Routing Table:752
Average APNIC Region AS path length visible:4.7
Max APNIC Region AS path length visible: 24
Number of APNIC region 32-bit ASNs visible in the Routing Table:239
Number of APNIC addresses announced to Internet:  703244928
Equivalent to 41 /8s, 234 /16s and 170 /24s
Percentage of available APNIC address space announced: 82.2

APNIC AS Blocks4608-4864, 7467-7722, 9216-10239, 17408-18431
(pre-ERX allocations)  23552-24575, 37888-38911, 45056-46079, 55296-56319,
   58368-59391, 131072-133119
APNIC Address Blocks 1/8,  14/8,  27/8,  36/8,  39/8,  42/8,  43/8,
49/8,  58/8,  59/8,  60/8,  61/8, 101/8, 103/8,
   106/8, 110/8, 111/8, 112/8, 113/8, 114/8, 115/8,
   116/8, 117/8, 118/8, 119/8, 120/8, 121/8, 122/8,
   123/8, 124/8, 125/8, 126/8, 133/8, 150/8, 153/8,
   163/8, 171/8, 175/8, 180/8, 182/8, 183/8, 202/8,
   203/8, 210/8, 211/8, 218/8, 219/8, 220/8, 221/8,
   222/8, 223/8,

ARIN Region Analysis Summary


Prefixes being announced by ARIN Region ASes:152291
Total ARIN prefixes after maximum aggregation:77306
ARIN Deaggregation factor: 1.97
Prefixes being announced from the ARIN address blocks:   153229
Unique aggregates announced from the ARIN address blocks: 68328
ARIN Region origin ASes present in the Internet Routing Table:15177
ARIN Prefixes per ASN:10.10
ARIN Region origin ASes announcing only 

Re: technical contact at ATT Wireless

[Tyler Haske]

On Fri, Jun 29, 2012 at 10:51 AM, Jared Mauch  wrote:
>
> On Jun 29, 2012, at 10:37 AM, Tyler Haske wrote:
>
>> I'm sorry you don't like it, and I know IPv6 will wash all this away
>> soon enough, but where I'm working we have no plans to implement IPv6,
>> or require our vendors/partners to readdress their networks to get a
>> VPN up.
>
> Just because there are no plans, this doesn't mean you shouldn't bring it up.
>
> Even if its a "radar"/"future" issue for their networking team, it does raise
> the profile in the asks with others.
>
> - Jared

Let it be known that I hate NAT with the burning passion of a million
suns. But I'm the junior in my workplace, and this is the advice of
the head honchos. I can easily see both sides of this. I would say
with a few implementations, (maybe 25 or fewer) NATing isn't that
difficult.

Granted we both know that NAT breaks basically everything and makes
troubleshooting a TON MORE FUN. But plenty of people out there (my
workplace included) would argue this till the cows come home.

Re: technical contact at ATT Wireless

[Jared Mauch]

On Jun 29, 2012, at 10:37 AM, Tyler Haske wrote:

> I'm sorry you don't like it, and I know IPv6 will wash all this away
> soon enough, but where I'm working we have no plans to implement IPv6,
> or require our vendors/partners to readdress their networks to get a
> VPN up.

Just because there are no plans, this doesn't mean you shouldn't bring it up.

Even if its a "radar"/"future" issue for their networking team, it does raise
the profile in the asks with others.

- Jared

Re: technical contact at ATT Wireless

[Tyler Haske]

> RFC1918 and VPN becomes non-scalable fast when you connect to lots of
> different organizations - it doesn't take long before two
> organizations you connect to both want to use 172.16.0.x/24 or
> 10.0.0.x/24 or 192.168.0.0/24, or similar).  The same logic goes for
> VPN clients - if one end is potentially using RFC1918, the other end
> better not use it.  Since you can usually only control one end of the
> VPN for road-warrior VPN, it's best to make that end not use RFC1918.
> Otherwise you may find you need to route 192.168.x.y, but that just so
> happens to be the user's default gateway, name server, printer, or
> other key device.  Of course having another set of NAT addresses for
> CGN will solve everything (yes, that's sarcastic, but I can predict
> how they'll be used to "solve" this type of problem).
>
> Just because "it usually works" doesn't mean using RFC1918 addresses
> for left and/or right subnets in a VPN is a good idea.

My workplace solves this by just NATing again. It isn't the best
solution but it does work. Put aside a 10.0.0.0/16 and whenever you
have a NATed network you want to connect a VPN to on the edge just
static NAT the addresses to make them unique again. Their 172.16.x.x
becomes your 10.2.x.x.

I dunno about 'not scalable'. I guess if your connecting to thousands
of networks it won't work well, but for a a few hundred it works well
enough.

I'm sorry you don't like it, and I know IPv6 will wash all this away
soon enough, but where I'm working we have no plans to implement IPv6,
or require our vendors/partners to readdress their networks to get a
VPN up.

Re: Constant low-level attack

[Rich Kulawiec]

On Thu, Jun 28, 2012 at 01:31:56PM -0700, Lou Katz wrote:
> 2. Is there anything useful to do with this info other than put the IP 
> addresses into a firewall reject table? 

Do you need to allow inbound ssh connections from the entire planet?

If not, then head over to ipdeny.com and grab the relevant network
allocations for the countries that you *do* need to allow them from.
Block everyone else, allow only the countries you need.

This won't solve your problem completely, but it'll take a substantial
bite out of it, and it'll minimize the number of additional point entries
that you need for annoying hosts whose connections originate in the
set of countries you need to allow.

Then: do you need to allow inbound ssh connections from all operating
systems?

If not, then use passive OS fingerprinting to block those which originate
from operating systems known not to be in use, particularly if those
operatng systems happen to be the ones running on a few hundred million
compromised systems.  (Obviously, this technique is far less effective
is you can't do that.  My condolences.)

And then: consider, instead of point blocks for the remaining annoyances,
use the enclosing /24.  A lot of compromised hosts are not on static
addresses, and guessing that they will bounce around inside (roughly)
a /24 is often a good enough approximation to reality that it works.
Your mileage may vary.

And then: scotch.  Macallan.  18-year.  You've earned it.

---rsk

Re: Constant low-level attack

[Alain Hebert]

Hi,

We implemented fail2ban about a year ago to cut down on incoming 
spamming (down from 500k+ emails a day to 20k)


Now what can I do with the ~11,000 IP's I identify as spammer every 
week :(


Reporting them to their Telco is pretty much a waste of time... 
they are not about to lose customers to something as trivial as computer 
security.


-
Alain Hebertaheb...@pubnix.net
PubNIX Inc.
50 boul. St-Charles
P.O. Box 26770 Beaconsfield, Quebec H9W 6G7
Tel: 514-990-5911  http://www.pubnix.netFax: 514-990-9443


On 06/28/12 17:52, TR Shaw wrote:

On Jun 28, 2012, at 4:31 PM, Lou Katz wrote:


The other day, I looked carefully at my auth.log (Xubuntu 11.04) and discovered 
many lines
of the form:

  Jun 28 13:13:54 localhost sshd[12654]: Bad protocol version 
identification '\200F\001\003\001' from 94.252.177.159

In the past day, I have recorded about 20,000 unique IP addresses used for this 
type of probe.
I doubt if this is a surprise to anyone - my question is twofold:

1. Does anyone want this evergrowing list of, I assume, compromised machines?
2. Is there anything useful to do with this info other than put the IP 
addresses into a firewall reject table? I have done
   that and do see a certain amount of repeat hits.

Just a note that if you were running fail2ban.org you would get automatic 
updates of your firewall and share the IPs with the community and get the 
advantage of the communities detections as well.