Re: [j-nsp] DOS Attack
* sherif mostafa: Could anyone help please as I've faced an error message DOS below that caused high CPU usage: ERROR 08/02/2010 16:22:46 CAI dosProtection: Flow is suspicious: GigabitEthernet11/0.410 for control protocol: IP TTL Expired source MAC 0018.742f.b380 with rate 241 pps Have you checked that you haven't got a routing loop or something like that? (And which platform is that, BTW?) If this isn't the case, you need to figure out who's got the 0018.742f.b380 MAC address and ask them to stop sending those packets. -- Florian Weimerfwei...@bfk.de BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße 100 tel: +49-721-96201-1 D-76133 Karlsruhe fax: +49-721-96201-99 ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: [j-nsp] DOS Attack
Dear Florian, This ERX, Administration of router interface 0018.742f.b380 belongs to me also, but should I filter all those packet types ?? To: sherifmka2...@hotmail.com CC: juniper-nsp@puck.nether.net Subject: Re: [j-nsp] DOS Attack From: fwei...@bfk.de Date: Wed, 4 Aug 2010 13:27:05 + * sherif mostafa: Could anyone help please as I've faced an error message DOS below that caused high CPU usage: ERROR 08/02/2010 16:22:46 CAI dosProtection: Flow is suspicious: GigabitEthernet11/0.410 for control protocol: IP TTL Expired source MAC 0018.742f.b380 with rate 241 pps Have you checked that you haven't got a routing loop or something like that? (And which platform is that, BTW?) If this isn't the case, you need to figure out who's got the 0018.742f.b380 MAC address and ask them to stop sending those packets. -- Florian Weimer fwei...@bfk.de BFK edv-consulting GmbH http://www.bfk.de/ Kriegsstraße 100 tel: +49-721-96201-1 D-76133 Karlsruhe fax: +49-721-96201-99 ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: [j-nsp] DOS Attack
-Original Message- From: juniper-nsp-boun...@puck.nether.net [mailto:juniper-nsp- boun...@puck.nether.net] On Behalf Of sherif mostafa Sent: Wednesday, August 04, 2010 9:37 AM To: fwei...@bfk.de Cc: juniper-nsp@puck.nether.net Subject: Re: [j-nsp] DOS Attack Dear Florian, This ERX, Administration of router interface 0018.742f.b380 belongs to me also, but should I filter all those packet types ?? Sounds like you have a routing loop, perhaps default routes on two neighbors pointing at each other? This is what I would suspect when I'm seeing TTL expirations. In any event, this error message is just telling you that is suspects that there is a problem, however are you actually observing poor performance or is it causing some type of outage or other issue? 241 pps really isn't that much - it's not what I would consider to be a normal flooding type of attack... Stefan Fouant, CISSP, JNCIEx2 www.shortestpathfirst.net GPG Key ID: 0xB5E3803D ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
[j-nsp] DOS Attack
Dears, Could anyone help please as I've faced an error message DOS below that caused high CPU usage: ERROR 08/02/2010 16:22:46 CAI dosProtection: Flow is suspicious: GigabitEthernet11/0.410 for control protocol: IP TTL Expired source MAC 0018.742f.b380 with rate 241 pps ERROR 08/02/2010 16:24:11 CAI dosProtection: Flow is suspicious: GigabitEthernet11/0.410 for control protocol: IP TTL Expired source MAC 0018.742f.b380 with rate 11 pps ERROR 08/02/2010 16:24:38 CAI dosProtection: Flow is suspicious: GigabitEthernet11/0.410 for control protocol: IP TTL Expired source MAC 0018.742f.b380 with rate 10 pps ERROR 08/02/2010 16:24:59 CAI dosProtection: Flow is suspicious: GigabitEthernet11/0.410 for control protocol: IP TTL Expired source MAC 0018.742f.b380 with rate 10 pps ERROR 08/02/2010 16:26:57 CAI dosProtection: Flow is suspicious: GigabitEthernet11/0.410 for control protocol: IP TTL Expired source MAC 0018.742f.b380 with rate 20 pps ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: [j-nsp] DOS attack?
Hi! I may open a JTAC case, when I understand for which problem I should open one. The following has happened: The system (J6350) is running under 9.3R2.8, so we decided to update to 9.4R2.8. That was a desaster, and I hope you can help also in this matter. After upgrading the systems, both thought they are VRRP master. The problem is obvious, two sstems thinking to be the same appliance. Was there a change in VRRP configuration? Because this was not working, we made a rollback. Now we have the same memory problems again. Starting the shell, bam Low memory, show bgp summary, los memory. Each event has consequences: disabling a BGP session. The system has 1 GB of DRAM. What are the memory intensive processes? The BGP sessions? We have a J6350 at the AMS-IX where are a lot more BGP sessions. The VRRP processes? The rpd process has a significant higher memory consumption than on the AMS-IX router for example. Before opening a JTAC ticket, you may be saying get 1 GB of memory, your problems will disappear. Regards, Matthias Am 17.05.2009 um 14:54 schrieb Richard A Steenbergen: On Sun, May 17, 2009 at 02:19:56AM -0700, Robert Raszuk wrote: Hi Matthias, I wonder now, which is the event, that triggered this behavious? The numer of ssh-logins at that time or this zbexpected EOF? I would with good deal of assurance conclude that the cause were ssh-login attack which apparently starved the poor box to it's memory limits. When even your kernel spins a panic message on the low of memory due to such attack control plane can exhibit quite unexpected behavior. In my opinion end-of-frame BGP message is just a consequence of this. The advice would be to: * open a case with jtac to find out why subsequent ssh-logins cause a memory leak * reduce to very max rate-limiting for the ssh logins It's probably more likely that the box was always getting floods of SSH connection attempts (because thats what happens to anything you leave on the Internet with port 22 open), and the low memory condition was coincidental or 99% caused by something else. The openssh people would be shitting bricks if there was actually a memory leak from multiple connection attempts. Filter thy lo0 (*), but my money is on a leak somewhere else (or someone running a 256mb RE :P). (*) I always found it weird that JUNOS lacks a software filter for access to things like SSH, like a line vty access-class on Crisco. Obviously a proper lo0 firewall filter is better, but there have been a few occasions where this has been problematic over the years (logical-routers pre-firewall split where the integration with policy was tricky, EX for the first year of its life where lo0 filters didn't work, etc). Something as simple as a prefix-list dumping to hosts.allow would probably be sufficient. Also being able to change the listen port to something other than 22 might help the people who for whatever reason aren't able to do a strict IP based filter (simple option under system services ssh to pass to sshd). -- Richard A Steenbergen r...@e-gerbil.net http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC) ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
[j-nsp] DOS attack?
Hi! Last night we had a mysterious behaviour on our router. On a BGP connection with Cogent we received an unexpected EOF. There were also a great number of SSH logins (we do not have FW rules in place, but we have a rate limit, Shortly after the router complained about low memory and a few BGP sessions drop down (oviosly the one, which are memory exhausting), I wonder now, which is the event, that triggered this behavious? The numer of ssh-logins at that time or this zbexpected EOF? The log of that time: May 17 04:29:24 emsdetten1 inetd[4291]: ssh from 82.165.235.170 exceeded counts/min (limit 10/min) May 17 04:29:25 emsdetten1 last message repeated 7 times May 17 04:29:36 emsdetten1 rpd[4303]: bgp_listen_accept: Connection attempt from unconfigured neighbor: 91.190.xxx.xxx+40432 May 17 04:29:52 emsdetten1 rpd[4303]: bgp_recv: peer 149.6.xxx.xxx (External AS 174): received unexpected EOF May 17 04:30:06 emsdetten1 rpd[4303]: bgp_listen_accept: Connection attempt from unconfigured neighbor: 91.190.xxx.xxx+43119 May 17 04:31:00 emsdetten1 /kernel: KERNEL_MEMORY_CRITICAL: System low on free memory, notifying init (#2). May 17 04:31:00 emsdetten1 cron[49326]: (root) CMD (adjkerntz -a) May 17 04:31:01 emsdetten1 rpd[4303]: Received low-memory signal: BGP Write active, 422 free pages May 17 04:31:01 emsdetten1 rpd[4303]: Processing low memory signal May 17 04:31:14 emsdetten1 rpd[4303]: bgp_listen_accept: Connection attempt from unconfigured neighbor: 193.108.xxx.xxx+52139 May 17 04:31:34 emsdetten1 /kernel: KERN_ARP_ADDR_CHANGE: arp info overwritten for 91.190.xxx.xxx from 00:00:1a:19:c1:0f to 00:00:1a: 19:c1:10 May 17 04:31:34 emsdetten1 sshd[49329]: Failed password for root from 82.165.235.170 port 56403 ssh2 May 17 04:31:34 emsdetten1 inetd[4291]: /usr/sbin/sshd[49329]: exited, status 255 May 17 04:31:35 emsdetten1 sshd[49331]: Failed password for root from 82.165.235.170 port 47707 ssh2 May 17 04:31:35 emsdetten1 inetd[4291]: /usr/sbin/sshd[49331]: exited, status 255 May 17 04:31:36 emsdetten1 sshd[49337]: Failed password for root from 82.165.235.170 port 57612 ssh2 May 17 04:31:36 emsdetten1 inetd[4291]: /usr/sbin/sshd[49337]: exited, status 255 May 17 04:31:36 emsdetten1 sshd[49339]: Failed password for root from 82.165.235.170 port 49046 ssh2 May 17 04:31:36 emsdetten1 rpd[4303]: bgp_listen_accept: Connection attempt from unconfigured neighbor: 91.190.xxx.xxx+47675 May 17 04:31:36 emsdetten1 inetd[4291]: /usr/sbin/sshd[49339]: exited, status 255 May 17 04:31:38 emsdetten1 sshd[49335]: Failed password for root from 82.165.235.170 port 38441 ssh2 May 17 04:31:38 emsdetten1 inetd[4291]: /usr/sbin/sshd[49335]: exited, status 255 May 17 04:31:39 emsdetten1 sshd[49330]: Failed password for root from 82.165.235.170 port 37700 ssh2 May 17 04:31:39 emsdetten1 inetd[4291]: /usr/sbin/sshd[49330]: exited, status 255 May 17 04:31:39 emsdetten1 sshd[49345]: Failed password for root from 82.165.235.170 port 40019 ssh2 May 17 04:31:39 emsdetten1 inetd[4291]: ssh from 82.165.235.170 exceeded counts/min (limit 10/min) May 17 04:31:40 emsdetten1 sshd[49343]: Failed password for root from 82.165.235.170 port 49411 ssh2 May 17 04:31:40 emsdetten1 inetd[4291]: /usr/sbin/sshd[49345]: exited, status 255 May 17 04:31:40 emsdetten1 inetd[4291]: ssh from 82.165.235.170 exceeded counts/min (limit 10/min) May 17 04:31:41 emsdetten1 sshd[49341]: Failed password for root from 82.165.235.170 port 57987 ssh2 May 17 04:31:41 emsdetten1 inetd[4291]: /usr/sbin/sshd[49341]: exited, status 255 May 17 04:31:41 emsdetten1 inetd[4291]: ssh from 82.165.235.170 exceeded counts/min (limit 10/min) May 17 04:31:41 emsdetten1 sshd[49347]: Failed password for root from 82.165.235.170 port 60041 ssh2 May 17 04:31:41 emsdetten1 inetd[4291]: /usr/sbin/sshd[49343]: exited, status 255 May 17 04:31:41 emsdetten1 inetd[4291]: /usr/sbin/sshd[49347]: exited, status 255 May 17 04:31:41 emsdetten1 inetd[4291]: ssh from 82.165.235.170 exceeded counts/min (limit 10/min) May 17 04:31:41 emsdetten1 last message repeated 6 times May 17 04:31:43 emsdetten1 rpd[4303]: bgp_listen_accept: Connection attempt from unconfigured neighbor: 193.108.xxx.xxx+49573 May 17 04:31:47 emsdetten1 inetd[4291]: ssh from 82.165.235.170 exceeded counts/min (limit 10/min) May 17 04:31:51 emsdetten1 sshd[49349]: Failed password for root from 218.26.118.106 port 49903 ssh2 May 17 04:31:52 emsdetten1 inetd[4291]: /usr/sbin/sshd[49349]: exited, status 255 May 17 04:31:52 emsdetten1 sshd[49351]: Failed password for root from 218.26.118.106 port 49931 ssh2 May 17 04:31:52 emsdetten1 inetd[4291]: /usr/sbin/sshd[49351]: exited, status 255 May 17 04:31:53 emsdetten1 inetd[4291]: ssh from 82.165.235.170 exceeded counts/min (limit 10/min) May 17 04:31:53 emsdetten1 last message repeated 2 times May 17 04:31:53 emsdetten1 rpd[4303]: Received
Re: [j-nsp] DOS attack?
Hi Matthias, I wonder now, which is the event, that triggered this behavious? The numer of ssh-logins at that time or this zbexpected EOF? I would with good deal of assurance conclude that the cause were ssh-login attack which apparently starved the poor box to it's memory limits. When even your kernel spins a panic message on the low of memory due to such attack control plane can exhibit quite unexpected behavior. In my opinion end-of-frame BGP message is just a consequence of this. The advice would be to: * open a case with jtac to find out why subsequent ssh-logins cause a memory leak * reduce to very max rate-limiting for the ssh logins Cheers, R. Hi! Last night we had a mysterious behaviour on our router. On a BGP connection with Cogent we received an unexpected EOF. There were also a great number of SSH logins (we do not have FW rules in place, but we have a rate limit, Shortly after the router complained about low memory and a few BGP sessions drop down (oviosly the one, which are memory exhausting), I wonder now, which is the event, that triggered this behavious? The numer of ssh-logins at that time or this zbexpected EOF? The log of that time: ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: [j-nsp] DOS attack?
The advice would be to: * open a case with jtac to find out why subsequent ssh-logins cause a memory leak * reduce to very max rate-limiting for the ssh logins Or even better - configure a firewall filter which limits ssh logins to your trusted netblocks - typically where your management stations etc are. Steinar Haug, Nethelp consulting, sth...@nethelp.no ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
Re: [j-nsp] DOS attack?
On Sun, May 17, 2009 at 02:19:56AM -0700, Robert Raszuk wrote: Hi Matthias, I wonder now, which is the event, that triggered this behavious? The numer of ssh-logins at that time or this zbexpected EOF? I would with good deal of assurance conclude that the cause were ssh-login attack which apparently starved the poor box to it's memory limits. When even your kernel spins a panic message on the low of memory due to such attack control plane can exhibit quite unexpected behavior. In my opinion end-of-frame BGP message is just a consequence of this. The advice would be to: * open a case with jtac to find out why subsequent ssh-logins cause a memory leak * reduce to very max rate-limiting for the ssh logins It's probably more likely that the box was always getting floods of SSH connection attempts (because thats what happens to anything you leave on the Internet with port 22 open), and the low memory condition was coincidental or 99% caused by something else. The openssh people would be shitting bricks if there was actually a memory leak from multiple connection attempts. Filter thy lo0 (*), but my money is on a leak somewhere else (or someone running a 256mb RE :P). (*) I always found it weird that JUNOS lacks a software filter for access to things like SSH, like a line vty access-class on Crisco. Obviously a proper lo0 firewall filter is better, but there have been a few occasions where this has been problematic over the years (logical-routers pre-firewall split where the integration with policy was tricky, EX for the first year of its life where lo0 filters didn't work, etc). Something as simple as a prefix-list dumping to hosts.allow would probably be sufficient. Also being able to change the listen port to something other than 22 might help the people who for whatever reason aren't able to do a strict IP based filter (simple option under system services ssh to pass to sshd). -- Richard A Steenbergen r...@e-gerbil.net http://www.e-gerbil.net/ras GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC) ___ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp
