[j-nsp] DOS attack?

2009-05-17 Thread Matthias Gelbhardt 

Hi!

Last night we had a mysterious behaviour on our router. On a BGP  
connection with Cogent we received an unexpected EOF. There were also  
a great number of SSH logins (we do not have FW rules in place, but we  
have a rate limit,  Shortly after the router complained about low  
memory and a few BGP sessions drop down (oviosly the one, which are  
memory exhausting),


I wonder now, which is the event, that triggered this behavious? The  
numer of ssh-logins at that time or this zbexpected EOF?


The log of that time:

May 17 04:29:24  emsdetten1 inetd[4291]: ssh from 82.165.235.170  
exceeded counts/min (limit 10/min)

May 17 04:29:25  emsdetten1 last message repeated 7 times
May 17 04:29:36  emsdetten1 rpd[4303]: bgp_listen_accept: Connection  
attempt from unconfigured neighbor: 91.190.xxx.xxx+40432
May 17 04:29:52  emsdetten1 rpd[4303]: bgp_recv: peer 149.6.xxx.xxx  
(External AS 174): received unexpected EOF
May 17 04:30:06  emsdetten1 rpd[4303]: bgp_listen_accept: Connection  
attempt from unconfigured neighbor: 91.190.xxx.xxx+43119
May 17 04:31:00  emsdetten1 /kernel: KERNEL_MEMORY_CRITICAL: System  
low on free memory, notifying init (#2).

May 17 04:31:00  emsdetten1 cron[49326]: (root) CMD (adjkerntz -a)
May 17 04:31:01  emsdetten1 rpd[4303]: Received low-memory signal: BGP  
Write active, 422 free pages

May 17 04:31:01  emsdetten1 rpd[4303]: Processing low memory signal
May 17 04:31:14  emsdetten1 rpd[4303]: bgp_listen_accept: Connection  
attempt from unconfigured neighbor: 193.108.xxx.xxx+52139
May 17 04:31:34  emsdetten1 /kernel: KERN_ARP_ADDR_CHANGE: arp info  
overwritten for 91.190.xxx.xxx from 00:00:1a:19:c1:0f to 00:00:1a: 
19:c1:10
May 17 04:31:34  emsdetten1 sshd[49329]: Failed password for root from  
82.165.235.170 port 56403 ssh2
May 17 04:31:34  emsdetten1 inetd[4291]: /usr/sbin/sshd[49329]:  
exited, status 255
May 17 04:31:35  emsdetten1 sshd[49331]: Failed password for root from  
82.165.235.170 port 47707 ssh2
May 17 04:31:35  emsdetten1 inetd[4291]: /usr/sbin/sshd[49331]:  
exited, status 255
May 17 04:31:36  emsdetten1 sshd[49337]: Failed password for root from  
82.165.235.170 port 57612 ssh2
May 17 04:31:36  emsdetten1 inetd[4291]: /usr/sbin/sshd[49337]:  
exited, status 255
May 17 04:31:36  emsdetten1 sshd[49339]: Failed password for root from  
82.165.235.170 port 49046 ssh2
May 17 04:31:36  emsdetten1 rpd[4303]: bgp_listen_accept: Connection  
attempt from unconfigured neighbor: 91.190.xxx.xxx+47675
May 17 04:31:36  emsdetten1 inetd[4291]: /usr/sbin/sshd[49339]:  
exited, status 255
May 17 04:31:38  emsdetten1 sshd[49335]: Failed password for root from  
82.165.235.170 port 38441 ssh2
May 17 04:31:38  emsdetten1 inetd[4291]: /usr/sbin/sshd[49335]:  
exited, status 255
May 17 04:31:39  emsdetten1 sshd[49330]: Failed password for root from  
82.165.235.170 port 37700 ssh2
May 17 04:31:39  emsdetten1 inetd[4291]: /usr/sbin/sshd[49330]:  
exited, status 255
May 17 04:31:39  emsdetten1 sshd[49345]: Failed password for root from  
82.165.235.170 port 40019 ssh2
May 17 04:31:39  emsdetten1 inetd[4291]: ssh from 82.165.235.170  
exceeded counts/min (limit 10/min)
May 17 04:31:40  emsdetten1 sshd[49343]: Failed password for root from  
82.165.235.170 port 49411 ssh2
May 17 04:31:40  emsdetten1 inetd[4291]: /usr/sbin/sshd[49345]:  
exited, status 255
May 17 04:31:40  emsdetten1 inetd[4291]: ssh from 82.165.235.170  
exceeded counts/min (limit 10/min)
May 17 04:31:41  emsdetten1 sshd[49341]: Failed password for root from  
82.165.235.170 port 57987 ssh2
May 17 04:31:41  emsdetten1 inetd[4291]: /usr/sbin/sshd[49341]:  
exited, status 255
May 17 04:31:41  emsdetten1 inetd[4291]: ssh from 82.165.235.170  
exceeded counts/min (limit 10/min)
May 17 04:31:41  emsdetten1 sshd[49347]: Failed password for root from  
82.165.235.170 port 60041 ssh2
May 17 04:31:41  emsdetten1 inetd[4291]: /usr/sbin/sshd[49343]:  
exited, status 255
May 17 04:31:41  emsdetten1 inetd[4291]: /usr/sbin/sshd[49347]:  
exited, status 255
May 17 04:31:41  emsdetten1 inetd[4291]: ssh from 82.165.235.170  
exceeded counts/min (limit 10/min)

May 17 04:31:41  emsdetten1 last message repeated 6 times
May 17 04:31:43  emsdetten1 rpd[4303]: bgp_listen_accept: Connection  
attempt from unconfigured neighbor: 193.108.xxx.xxx+49573
May 17 04:31:47  emsdetten1 inetd[4291]: ssh from 82.165.235.170  
exceeded counts/min (limit 10/min)
May 17 04:31:51  emsdetten1 sshd[49349]: Failed password for root from  
218.26.118.106 port 49903 ssh2
May 17 04:31:52  emsdetten1 inetd[4291]: /usr/sbin/sshd[49349]:  
exited, status 255
May 17 04:31:52  emsdetten1 sshd[49351]: Failed password for root from  
218.26.118.106 port 49931 ssh2
May 17 04:31:52  emsdetten1 inetd[4291]: /usr/sbin/sshd[49351]:  
exited, status 255
May 17 04:31:53  emsdetten1 inetd[4291]: ssh from 82.165.235.170  
exceeded counts/min (limit 10/min)

May 17 04:31:53  emsdetten1 last message repeated 2 times
May 17 04:31:53  emsdetten1 rpd[4303]: Received

Re: [j-nsp] DOS attack?

2009-05-17 Thread Robert Raszuk 

Hi Matthias,

 I wonder now, which is the event, that triggered this behavious? The
 numer of ssh-logins at that time or this zbexpected EOF?

I would with good deal of assurance conclude that the cause were 
ssh-login attack which apparently starved the poor box to it's memory 
limits.


When even your kernel spins a panic message on the low of memory due to 
such attack control plane can exhibit quite unexpected behavior. In my 
opinion end-of-frame BGP message is just a consequence of this.


The advice would be to:

* open a case with jtac to find out why subsequent ssh-logins cause a 
memory leak


* reduce to very max rate-limiting for the ssh logins

Cheers,
R.



Hi!

Last night we had a mysterious behaviour on our router. On a BGP 
connection with Cogent we received an unexpected EOF. There were also a 
great number of SSH logins (we do not have FW rules in place, but we 
have a rate limit,  Shortly after the router complained about low memory 
and a few BGP sessions drop down (oviosly the one, which are memory 
exhausting),


I wonder now, which is the event, that triggered this behavious? The 
numer of ssh-logins at that time or this zbexpected EOF?


The log of that time:


___
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Re: [j-nsp] DOS attack?

2009-05-17 Thread sthaug 
 The advice would be to:
 
 * open a case with jtac to find out why subsequent ssh-logins cause a 
 memory leak
 
 * reduce to very max rate-limiting for the ssh logins

Or even better - configure a firewall filter which limits ssh logins
to your trusted netblocks - typically where your management stations
etc are.

Steinar Haug, Nethelp consulting, sth...@nethelp.no
___
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp

Re: [j-nsp] DOS attack?

2009-05-17 Thread Richard A Steenbergen 
On Sun, May 17, 2009 at 02:19:56AM -0700, Robert Raszuk wrote:
 Hi Matthias,
 
  I wonder now, which is the event, that triggered this behavious? The
  numer of ssh-logins at that time or this zbexpected EOF?
 
 I would with good deal of assurance conclude that the cause were 
 ssh-login attack which apparently starved the poor box to it's memory 
 limits.
 
 When even your kernel spins a panic message on the low of memory due to 
 such attack control plane can exhibit quite unexpected behavior. In my 
 opinion end-of-frame BGP message is just a consequence of this.
 
 The advice would be to:
 
 * open a case with jtac to find out why subsequent ssh-logins cause a 
 memory leak
 
 * reduce to very max rate-limiting for the ssh logins

It's probably more likely that the box was always getting floods of SSH
connection attempts (because thats what happens to anything you leave on
the Internet with port 22 open), and the low memory condition was 
coincidental or 99% caused by something else. The openssh people would 
be shitting bricks if there was actually a memory leak from multiple 
connection attempts. Filter thy lo0 (*), but my money is on a leak 
somewhere else (or someone running a 256mb RE :P).

(*) I always found it weird that JUNOS lacks a software filter for
access to things like SSH, like a line vty access-class on Crisco. 
Obviously a proper lo0 firewall filter is better, but there have been
a few occasions where this has been problematic over the years
(logical-routers pre-firewall split where the integration with policy
was tricky, EX for the first year of its life where lo0 filters didn't
work, etc). Something as simple as a prefix-list dumping to hosts.allow
would probably be sufficient. Also being able to change the listen port
to something other than 22 might help the people who for whatever reason
aren't able to do a strict IP based filter (simple option under system
services ssh to pass to sshd).

-- 
Richard A Steenbergen r...@e-gerbil.net   http://www.e-gerbil.net/ras
GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
___
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp