RE: Level3 routing issues?
Of the customers I've had to shut off for being DOS targets, all are windows boxen. Perhaps there is a new windows exploit? Regards, Christopher J. Wolff, VP CIO Broadband Laboratories, Inc. http://www.bblabs.com -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of hc Sent: Friday, January 24, 2003 11:39 PM To: Joel Perez Cc: Aaron Burnett; Alex Rubenstein; [EMAIL PROTECTED] Subject: Re: Level3 routing issues? Okay this is getting bad.. one of our routers just locked up from udp 1434's. Can't even telnet to it now. -hc Joel Perez wrote: My firewalls are going nuts with hits on UDP port 1434 also from everywhere! -Original Message- From: Aaron Burnett [mailto:[EMAIL PROTECTED]] Sent: Sat 1/25/2003 1:19 AM To: Alex Rubenstein Cc: hc; [EMAIL PROTECTED] Subject: Re: Level3 routing issues? On Sat, 25 Jan 2003, Alex Rubenstein wrote: I dunno about that. But, I am seeing, in the last couple hours, all kinds of new traffic. like, customers who never get attacked or anything, all of a sudden: http://mrtg.nac.net/switch9.oct.nac.net/3865/switch9.oct.nac.net-3865.ht ml We are seeing this on ports all across out network -- nearly 1/2 our ports are in delta alarm right now. Anyone else? Yep. Since about 12:30 am. Getting pounded on UDP port 1434 from all over the world to any address on my network.
DOS?
Greetings, It looks like all hell is breaking loose on some of the nations backbones. http://www.internethealthreport.com The port counters on my ATT DS3 were reading in the 250 megabit range, that is a DS3, mind you. Any source IP's I can add to the circular file would be appreciated. Any ranges I find I'll echo back to the list. Regards, Christopher J. Wolff, VP CIO Broadband Laboratories, Inc. http://www.bblabs.com
Re: New worm / port 1434?
* Avleen Vig [EMAIL PROTECTED] [20030124 22:44]:
It seems we have a new worm hitting Microsoft SQL server servers on port
1434.
A preliminary look at some of our NetFlow data shows a suspect ICMP payload
delivered to one of our downstream colo customer boxes followed by a
70 Mbit/s burst from them. The burst consisted of traffic to seemingly random
destinations on 1434/udp. This customer typically does about 0.250 Mbit/s
so this was a bit out of their profile. :-) Needless to say, we shut them
down per a suspected security incident. The ICMP came from 66.214.194.31
though that could quite easily be forged or just another compromised box.
We're seeing red to many networks all over the world though our network seems
to have quieted down a bit. Sounds like a DDoS in the works.
Anyone else able to corroborate/compare notes?
-jr
Josh Richards jrichard@{ geekresearch.com, cubicle.net, digitalwest.net }
Geek Research, LLC - Digital West Networks, Inc - San Luis Obispo, CA
KG6CYK - IP/Unix/telecom/knowledge/coffee/security/crypto/business/geek
fyi (fwd)
-- Alex Rubenstein, AR97, K2AHR, [EMAIL PROTECTED], latency, Al Reuben -- --Net Access Corporation, 800-NET-ME-36, http://www.nac.net -- -- Forwarded message -- Date: Sat, 25 Jan 2003 01:50:34 -0500 From: Tim Yocum [EMAIL PROTECTED] To: [EMAIL PROTECTED] [EMAIL PROTECTED] Subject: fyi Might pass this along to nanog@ or anyone who cares to dig a bit deeper. I'm not subscribed to the list. Same packet structure, same length, different source port, always udp/1434. 0:48:25.375117 62.216.151.17.4322 unknown.Level3.net.ms-sql-m: [udp sum ok] udp 376 (ttl 117, id 52118, len 404) 0x 4500 0194 cb96 7511 ebef 3ed8 9711E...u.. 0x0010 d1f7 e4f1 10e2 059a 0180 9796 0401 0101 0x0020 0101 0101 0101 0101 0101 0101 0101 0101 0x0030 0101 0101 0101 0101 0101 0101 0101 0101 0x0040 0101 0101 0101 0101 0101 0101 0101 0101 0x0050 0101 0101 0101 0101 0101 0101 0101 0101 0x0060 0101 0101 0101 0101 0101 0101 0101 0101 0x0070 0101 0101 0101 0101 0101 0101 01dc c9b0 0x0080 42eb 0e01 0101 0101 0101 70ae 4201 70aeB.p.B.p. 0x0090 4290 9090 9090 9090 9068 dcc9 b042 b801Bh...B.. 0x00a0 0101 0131 c9b1 1850 e2fd 3501 0101 0550...1...P..5P 0x00b0 89e5 5168 2e64 6c6c 6865 6c33 3268 6b65..Qh.dllhel32hke 0x00c0 726e 5168 6f75 6e74 6869 636b 4368 4765rnQhounthickChGe 0x00d0 7454 66b9 6c6c 5168 3332 2e64 6877 7332tTf.llQh32.dhws2 0x00e0 5f66 b965 7451 6873 6f63 6b66 b974 6f51_f.etQhsockf.toQ 0x00f0 6873 656e 64be 1810 ae42 8d45 d450 ff16hsendB.E.P.. 0x0100 508d 45e0 508d 45f0 50ff 1650 be10 10aeP.E.P.E.P..P 0x0110 428b 1e8b 033d 558b ec51 7405 be1c 10aeB=U..Qt. 0x0120 42ff 16ff d031 c951 5150 81f1 0301 049bB1.QQP.. 0x0130 81f1 0101 0101 518d 45cc 508b 45c0 50ff..Q.E.P.E.P. 0x0140 166a 116a 026a 02ff d050 8d45 c450 8b45.j.j.j...P.E.P.E 0x0150 c050 ff16 89c6 09db 81f3 3c61 d9ff 8b45.Pa...E 0x0160 b48d 0c40 8d14 88c1 e204 01c2 c1e2 0829...@...) 0x0170 c28d 0490 01d8 8945 b46a 108d 45b0 5031...E.j..E.P1 0x0180 c951 6681 f178 0151 8d45 0350 8b45 ac50.Qf..x.Q.E.P.E.P 0x0190 ffd6 ebca
RE: Level3 routing issues?
Not just L3Genuity is getting whacked. ELI is getting whacked. Somebody needs to be gelded. Andrew
Re: Level3 routing issues?
This is definately a world-wide problem. Many networks are reporting all sorts of things. Nothing clear, except that it's all aimed at 1434. 01:28:33.331686 64.21.34.210.28295 238.192.142.61.1434: udp 376 [ttl 1] 01:28:33.331720 207.99.21.121.1917 226.39.19.228.1434: udp 376 [ttl 1] 01:28:33.331772 64.247.0.168.1379 239.194.46.210.1434: udp 376 [ttl 1] 01:28:33.331841 207.99.77.34.3894 227.154.8.29.1434: udp 376 [ttl 1] 01:28:33.331992 207.99.21.120.2558 231.16.91.78.1434: udp 376 [ttl 1] FYI: ms-sql-m1434/tcp #Microsoft-SQL-Monitor ms-sql-m1434/udp #Microsoft-SQL-Monitor On Sat, 25 Jan 2003, hc wrote: I am on Verizon-GNI via Qwest and Genuity and seeing the same problem as well. -hc Joel Perez wrote: I am also seeing increased traffic on my network. It has gotten so bad for one of my edge routers that i cant telnet into it. But i am on Qwest and GBLX. -Original Message- From: Alex Rubenstein [mailto:[EMAIL PROTECTED]] Sent: Sat 1/25/2003 1:04 AM To: hc Cc: [EMAIL PROTECTED] Subject: Re: Level3 routing issues? I dunno about that. But, I am seeing, in the last couple hours, all kinds of new traffic. like, customers who never get attacked or anything, all of a sudden: http://mrtg.nac.net/switch9.oct.nac.net/3865/switch9.oct.nac.net-3865.html We are seeing this on ports all across out network -- nearly 1/2 our ports are in delta alarm right now. Anyone else? I will dig more to look at the traffic. On Sat, 25 Jan 2003, hc wrote: Anyone seeing routing problems with Level3 at this hour? I just witnessed tons of prefixes behind level3's network withdraw. Any information on what is happening (if you know) would be great. Thanks! -hc -- Alex Rubenstein, AR97, K2AHR, [EMAIL PROTECTED], latency, Al Reuben -- --Net Access Corporation, 800-NET-ME-36, http://www.nac.net -- -- Alex Rubenstein, AR97, K2AHR, [EMAIL PROTECTED], latency, Al Reuben -- --Net Access Corporation, 800-NET-ME-36, http://www.nac.net --
Re: DOS?
On 1/25/03 2:00 AM, Christopher J. Wolff [EMAIL PROTECTED] wrote:
Greetings,
It looks like all hell is breaking loose on some of the nations
backbones. http://www.internethealthreport.com
The port counters on my ATT DS3 were reading in the 250 megabit range,
that is a DS3, mind you.
Any source IP's I can add to the circular file would be appreciated.
Any ranges I find I'll echo back to the list.
Regards,
Christopher J. Wolff, VP CIO
Broadband Laboratories, Inc.
http://www.bblabs.com
You need a filter similar to this (in junos format):
show configuration firewall filter filter-012503
term deny-dos {
from {
packet-length 404;
protocol udp;
destination-port 1434;
}
then {
count codered-4;
discard;
}
}
term allow-rest {
then accept;
}
--Phil
ISPrime
Re: Level3 routing issues?
Internap has posted an alert noting widespread latency and packetloss affecting all their pnaps. Any SQL Server host at my facilily shows an enourmous traffic spike at the times below. We've begun filtering udp port 1434 in/out. - Original Message - From: Andy Dills [EMAIL PROTECTED] To: Alex Rubenstein [EMAIL PROTECTED] Cc: hc [EMAIL PROTECTED]; [EMAIL PROTECTED] Sent: Friday, January 24, 2003 10:37 PM Subject: Re: Level3 routing issues? On Sat, 25 Jan 2003, Alex Rubenstein wrote: I dunno about that. But, I am seeing, in the last couple hours, all kinds of new traffic. like, customers who never get attacked or anything, all of a sudden: http://mrtg.nac.net/switch9.oct.nac.net/3865/switch9.oct.nac.net-3865.html We are seeing this on ports all across out network -- nearly 1/2 our ports are in delta alarm right now. Anyone else? I will dig more to look at the traffic. Interesting, at almost the exact same time (call it 12:30), qwest dropped all but 1000 routes through IAD...still trying to get somebody on the phone at their IP noc, not having much luck. Genuity seems fine at the moment... Any speculation yet? Kind of an odd coincidence of problems... Oh, just got through...fiber cut in DC? Andy Andy Dills 301-682-9972 Xecunet, LLCwww.xecu.net Dialup * Webhosting * E-Commerce * High-Speed Access
Re: Level3 routing issues?
Really, really bad - most traffic I see is from this virus/dos: Extended IP access list 152 deny udp any any eq 1434 (5639464 matches) - 94% permit ip any any (311888 matches) - 6% Wow!!! On Fri, 24 Jan 2003 [EMAIL PROTECTED] wrote: Really bad. Quick capture of filter drops: PROTO 17 (UDP) pkt from (IP's from all over the world)/1033 to (All my IP space)/1434 dropped On Sat, 25 Jan 2003, hc wrote: Okay this is getting bad.. one of our routers just locked up from udp 1434's. Can't even telnet to it now. -hc Joel Perez wrote: My firewalls are going nuts with hits on UDP port 1434 also from everywhere! -Original Message- From: Aaron Burnett [mailto:[EMAIL PROTECTED]] Sent: Sat 1/25/2003 1:19 AM To: Alex Rubenstein Cc: hc; [EMAIL PROTECTED] Subject: Re: Level3 routing issues? On Sat, 25 Jan 2003, Alex Rubenstein wrote: I dunno about that. But, I am seeing, in the last couple hours, all kinds of new traffic. like, customers who never get attacked or anything, all of a sudden: http://mrtg.nac.net/switch9.oct.nac.net/3865/switch9.oct.nac.net-3865.html We are seeing this on ports all across out network -- nearly 1/2 our ports are in delta alarm right now. Anyone else? Yep. Since about 12:30 am. Getting pounded on UDP port 1434 from all over the world to any address on my network.
New worm/DOS/Level3 routing issues
repost* Forgive me if this shows up twice. Mail is flaked via this smtp, and the last time I sent this, I accidentally sent it to the individual and not list. heh. Temporary block in place. My border cpu was starting to hammer up. Outbound stat about 2 minutes later: deny udp any any eq 1434 (445523 matches) permit ip 69.8.0.0 0.0.63.255 any (55749 matches) permit ip 206.27.138.0 0.0.1.255 any permit ip 206.30.96.0 0.0.31.255 any (97851 matches) permit ip 205.162.224.0 0.0.15.255 any (146920 matches) permit ip 205.240.128.0 0.0.15.255 any (49146 matches) permit ip 204.249.192.0 0.0.15.255 any (27351 matches) permit ip 192.133.7.0 0.0.0.255 any (5 matches) permit ip 63.136.128.0 0.0.3.255 any (379 matches) permit ip 216.226.0.0 0.0.31.255 any (27173 matches) permit ip 64.58.32.0 0.0.15.255 any (17368 matches) permit ip 206.230.34.128 0.0.0.127 any permit ip 209.54.40.0 0.0.1.255 any permit ip 206.61.140.0 0.0.0.255 any (52 matches) Inbound stat at same time: deny udp any any eq 1434 (53534 matches) permit ip any any (431556 matches) cpu load drop of about 20%Definately a bad port. virus suspected due to inbound and outbound. Jack Bates Network Engineer BrightNet Oklahoma
Re: New worm / port 1434?
We were hit hard by this as well. It appears to be a buffer overflow exploit, as blocking the ports on my router and restarting MS SQL put a stop to it. Thanks, Adam Debus Network Administrator, ReachONE Internet [EMAIL PROTECTED] - Original Message - From: Avleen Vig [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Friday, January 24, 2003 10:32 PM Subject: New worm / port 1434? It seems we have a new worm hitting Microsoft SQL server servers on port 1434.
RE: Level3 routing issues?
We are also seeing this traffic at AS4436. Appears to be coming from IP addresses all over the space. Here's a box that traps all of 165.227.0.0/16: 23:08:13.257197 165.194.123.131.1227 165.227.92.176.1434: udp 376 23:08:13.259778 129.187.150.78.2667 165.227.84.186.1434: udp 376 23:08:13.276695 61.40.143.242.3794 165.227.21.48.1434: udp 376 23:08:13.284191 128.218.133.213.1078 165.227.198.96.1434: udp 376 23:08:13.286648 169.229.141.44.1065 165.227.255.90.1434: udp 376 23:08:13.294512 218.232.109.22.3302 165.227.146.129.1434: udp 376 23:08:13.300412 137.79.10.100.2478 165.227.5.230.1434: udp 376 23:08:13.302869 128.143.100.86.1397 165.227.41.248.1434: udp 376 23:08:13.317327 203.226.64.220.3081 165.227.216.188.1434: udp 376 23:08:13.319908 209.41.170.8.4033 165.227.252.85.1434: udp 376 23:08:13.322365 64.71.177.201.2439 165.227.128.21.1434: udp 376 23:08:13.327937 216.120.60.154.3005 165.227.125.156.1434: udp 376 23:08:13.330435 64.239.145.3.3231 165.227.4.161.1434: udp 376 23:08:13.333016 204.228.229.106.4049 165.227.238.69.1434: udp 376 23:08:13.335350 212.209.231.186.52703 165.227.38.136.1434: udp 376 23:08:13.337930 207.46.200.162.2343 165.227.96.170.1434: udp 376 23:08:13.340388 61.178.83.30.4525 165.227.77.119.1434: udp 376 23:08:13.342887 62.250.16.28.1385 165.227.119.91.1434: udp 376 23:08:13.345468 66.155.116.10.1041 165.227.106.35.1434: udp 376 23:08:13.362506 207.226.255.124.2331 165.227.189.42.1434: udp 376 23:08:13.364964 63.241.139.196.1150 165.227.135.221.1434: udp 376 23:08:13.367422 66.109.239.200.1117 165.227.67.250.1434: udp 376 23:08:13.370042 194.100.187.36.2342 165.227.103.27.1434: udp 376 23:08:13.372501 158.38.141.86.3269 165.227.239.113.1434: udp 376 23:08:13.374959 212.71.66.23.2019 165.227.232.118.1434: udp 376 23:08:13.377417 158.38.141.65.1382 165.227.169.58.1434: udp 376 23:08:13.379915 130.127.8.157.2980 165.227.107.122.1434: udp 376 23:08:13.382496 207.46.200.146.2718 165.227.49.107.1434: udp 376 23:08:13.386100 80.237.200.171.1198 165.227.93.216.1434: udp 376 23:08:13.388557 64.71.180.135.1915 165.227.38.41.1434: udp 376 23:08:13.394660 211.117.60.188.2806 165.227.49.12.1434: udp 376 -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Scott Granados Sent: Friday, January 24, 2003 10:41 PM To: Alex Rubenstein Cc: hc; [EMAIL PROTECTED] Subject: Re: Level3 routing issues? We just had a box inside one of my customers networks start sending tons of small packets not sure what kind yet. On Sat, 25 Jan 2003, Alex Rubenstein wrote: I dunno about that. But, I am seeing, in the last couple hours, all kinds of new traffic. like, customers who never get attacked or anything, all of a sudden: http://mrtg.nac.net/switch9.oct.nac.net/3865/s witch9.oct.nac.net-3865. html We are seeing this on ports all across out network -- nearly 1/2 our ports are in delta alarm right now. Anyone else? I will dig more to look at the traffic. On Sat, 25 Jan 2003, hc wrote: Anyone seeing routing problems with Level3 at this hour? I just witnessed tons of prefixes behind level3's network withdraw. Any information on what is happening (if you know) would be great. Thanks! -hc -- Alex Rubenstein, AR97, K2AHR, [EMAIL PROTECTED], latency, Al Reuben -- --Net Access Corporation, 800-NET-ME-36, http://www.nac.net --
Re: Level3 routing issues?
Hey Blaine, On Sat, Jan 25, 2003 at 01:53:49AM -0600, Blaine Kahle wrote: Same symptoms here. After disabling MS SQL, which required a reboot as the process didn't want to shut down normally, the traffic stopped. I found 3 boxes on our network that were generating massive amounts of traffic, all of which run MS SQL. This may or may not prove useful: http://www.microsoft.com/Downloads/details.aspx?displaylang=enFamilyID=DCFDCBE9-B4EB-4446-9BE7-2DE45CFA6A89 Cheers, --Adam -- Adam Korab
Re: Level3 routing issues?
From: Dave Stewart Lots of traffic on udp port 1434 coming in here via TW Telecom and Sprint Looks like we may have a winner for DDoS of the year (so far) Temporary block in place. My border cpu was starting to hammer up. Outbound stat about 2 minutes later: deny udp any any eq 1434 (445523 matches) permit ip 69.8.0.0 0.0.63.255 any (55749 matches) permit ip 206.27.138.0 0.0.1.255 any permit ip 206.30.96.0 0.0.31.255 any (97851 matches) permit ip 205.162.224.0 0.0.15.255 any (146920 matches) permit ip 205.240.128.0 0.0.15.255 any (49146 matches) permit ip 204.249.192.0 0.0.15.255 any (27351 matches) permit ip 192.133.7.0 0.0.0.255 any (5 matches) permit ip 63.136.128.0 0.0.3.255 any (379 matches) permit ip 216.226.0.0 0.0.31.255 any (27173 matches) permit ip 64.58.32.0 0.0.15.255 any (17368 matches) permit ip 206.230.34.128 0.0.0.127 any permit ip 209.54.40.0 0.0.1.255 any permit ip 206.61.140.0 0.0.0.255 any (52 matches) Inbound stat at same time: deny udp any any eq 1434 (53534 matches) permit ip any any (431556 matches) cpu load drop of about 20%Definately a bad port. virus suspected due to inbound and outbound. Jack Bates Network Engineer BrightNet Oklahoma
Re: Level3 routing issues?
MS SQL, or SQL Monitor? On Sat, 25 Jan 2003, Blaine Kahle wrote: On Sat, Jan 25, 2003 at 02:05:42AM -0500, Kevin Welch wrote: I am seeing similar traffic loads on my network at this hour, one of our MS SQL servers seemed to be sending a large amount of traffic out to the Internet. Still looking into it but too similar for me to avoid sending an e-mail. Same symptoms here. After disabling MS SQL, which required a reboot as the process didn't want to shut down normally, the traffic stopped. I found 3 boxes on our network that were generating massive amounts of traffic, all of which run MS SQL. -- Blaine Kahle [EMAIL PROTECTED] 0x178AA0E0 -- Alex Rubenstein, AR97, K2AHR, [EMAIL PROTECTED], latency, Al Reuben -- --Net Access Corporation, 800-NET-ME-36, http://www.nac.net --
Re: New worm / port 1434?
At 02:45 AM 1/25/2003 -0600, Jack Bates wrote: From: Mike Tancsa Yes, I am seeing this big time. Are you sure its SQL server ? Thats normally 1433 no ? Are there any other details somewhere about this ? snip All MS SQL servers listen to 1434 reguardless of the other ports they listen on. Depending on configuration depends on what other ports it uses (due to various security models), but 1434 is a constant in all configurations according to a quick search and a read on the last MS SQL vulnerability found in 7/2002. Thanks, I have blocked the infected hosts in my customer colo space. Its an eye opener how much traffic they generate on the local collision domain they are on :-( ---Mike Mike Tancsa, tel +1 519 651 3400 Sentex Communications, [EMAIL PROTECTED] Providing Internet since 1994www.sentex.net Cambridge, Ontario Canada www.sentex.net/mike
Re: DOS?
On Sat, 25 Jan 2003, Christopher J. Wolff wrote: Greetings, It looks like all hell is breaking loose on some of the nations backbones. http://www.internethealthreport.com The port counters on my ATT DS3 were reading in the 250 megabit range, that is a DS3, mind you. Any source IP's I can add to the circular file would be appreciated. Any ranges I find I'll echo back to the list. It's an MS SQL worm that is sending and receiving UDP on 1434. http://www.nextgenss.com/advisories/mssql-udp.txt appears to be relevant. Anyone want to get involved in some sort of real time chat (like IRC) to disuss strategies? We're seeing some pretty big traffic, and related problems in multiple colo's world wide. Doug -- We have known freedom's price. We have shown freedom's power. And in this great conflict, ... we will see freedom's victory. - George W. Bush, President of the United States State of the Union, January 28, 2002 Do YOU Yahoo!?
Re: New worm / port 1434?
Duplicated info.. But this is an old worm ;-( http://www.cert.org/advisories/CA-1996-01.html Pete Ashdown wrote: * Avleen Vig ([EMAIL PROTECTED]) [030124 23:50] writeth: It seems we have a new worm hitting Microsoft SQL server servers on port 1434. Affirmative. Be sure to block 1434 UDP on both the inbound and the outbound. Infected servers are VERY NOISY. -- Message scanned for viruses and dangerous content by http://www.newnet.co.uk/av/ and believed to be clean
SQL Server Worm?
I can't say for certain, not having taken an exhaustive look (it is, after all, almost 3 in the morning out here on the right coast), but on the one MS SQL server here, there do not appear to be new files installed, and after rebooting, the server is *not* spewing forth traffic as it was earlier. I don't see any new stored procedures at a glance, so perhaps rebooting resolves the issue. Of course: access-list xxx deny udp any any eq 1434 helps, as well, in my case... lotta incoming hits to 1434... but none outbound in the past 20 minutes
1434 traffic
What I'm seeing from on my personal network connections is a lot of traffic to udp port 1434 start at 05:30:08 UTC. The sources appear very widespread, but I'm also seeing different affects on networks. Some backbones are being hit extremely hard, while others are just moderately impacted. I haven't figured out if it is a customer base difference, or if the worm is targetting. I haven't been willing to sacrafice one of my personal computers to the cause, so I don't know what's in the payload. According to Matrix Systems, there was about a 10% drop over the next 30 minutes. Keynote's data shows several backbones impacted. BGP and DNS appear to be holding up more or less, but g.root-servers.net has left the building (may be self-inflected withdrawal). Cable Wireless's sla.cw.net show no impact on their network. UUNET's network status web site says Normal. Earthlink's network status web site shows various maintenance activity. SBC's network status web site says dial and dsl is Impaired. I can't reach www.sprint.net. ATT's network status is unavailable while service enhancement is being performed.
RE: Level3 routing issues?
Same results here, shut down SQL problem went away... started it back up.. problem started again, so I shut them all down. One side note all the egress traffic headed out UU.NET, not our CW or Sprint DS3's... since we have full routes from all carriers this may be an indicator of the destination. Too bad I have a 700MB netflow file I cannot load or parse or I might be able to provide more detailed information as to a destination. - Kevin Welch [EMAIL PROTECTED] Network EngineerThe Iserv Company Desk Ph: 616.493.0577 Cell Ph: 616.437.3861 -Original Message- From: Blaine Kahle [mailto:[EMAIL PROTECTED]] Sent: Saturday, January 25, 2003 2:54 AM To: Kevin Welch Cc: 'Alex Rubenstein'; 'hc'; [EMAIL PROTECTED] Subject: Re: Level3 routing issues? On Sat, Jan 25, 2003 at 02:05:42AM -0500, Kevin Welch wrote: I am seeing similar traffic loads on my network at this hour, one of our MS SQL servers seemed to be sending a large amount of traffic out to the Internet. Still looking into it but too similar for me to avoid sending an e-mail. Same symptoms here. After disabling MS SQL, which required a reboot as the process didn't want to shut down normally, the traffic stopped. I found 3 boxes on our network that were generating massive amounts of traffic, all of which run MS SQL. -- Blaine Kahle [EMAIL PROTECTED] 0x178AA0E0
Re: Level3 routing issues?
* Josh Richards [EMAIL PROTECTED] [20030124 23:25]:
Same here. We first saw what looked like a DoS at about
09:00 PST. We're seeing strange stuff all over the place.
Oops, meant to say 09:30 PST.
-jr
Josh Richards jrichard@{ geekresearch.com, cubicle.net, digitalwest.net }
Geek Research, LLC - Digital West Networks, Inc - San Luis Obispo, CA
KG6CYK - IP/Unix/telecom/knowledge/coffee/security/crypto/business/geek
Re: Level3 routing issues?
From: Mikael Abrahamsson What kind of traffic levels are you seeing? With a handful of /16 etc we're not seeing more than 5-10 megabits of traffic according to my global transit graphs. People who havent null routed their unused prefixes properly will probably see a lot of problems though (but that's default). Going by the decline in both my outbound and inbound access lists over time, I suspect that the traffic increases when a sql server is found. Once communication is cut between the two, it appears that there is just scan data passing through at a lower rate. I have little data to go on, though, so my assessment may not be accurate. Jack Bates BrightNet Oklahoma
New MS SQL Exploit DOS Attack started tonight at 12:30AM EST (GMT -0500)
Everyone, I don't know what is causing this, but we had several customer machines (which we don't manage) affected tonight. The common thread is that all were running an unpatched MS SQL Server. This new worm seems to create MASSIVE network traffic which propagates outbound. Somehow it seems to be amplified at each of our Cisco routers. In our colo facility, we had 3 infected servers on 10Base-T connections - after this traffic hit our core router, the traffic increased from just under 30Mbits/sec inbound from our colo switch to 80+Mbits/sec outbound over ALL transit and peering connections. I know our routers aren't smurf amplifiers and I don't know what caused the increased outbound traffic. Once this process is started, the MSSQLServer service cannot be stopped (or killed with pview). If the service is disabled and the server rebooted, it will not generate this traffic. It is not a master-slave program which requires a connection from outside to start the flow. Once the SQL server has been infected, no Internet connection is needed to continue the traffic storm even after a reboot. None of our managed customer machines were affected, but all of them are patched with current patches and none of them have 1433 exposed to the world either. I don't have any more detail at this time, but I plan to look into this worm/virus/exploit further in the AM. This seems to affect both MSSQL and MSDE. Does anyone else have more to add. I have seen several networks drop off the earth tonight as a result of this exploit. -Robert Tellurian Networks - The Ultimate Internet Connection http://www.tellurian.com | 888-TELLURIAN | 973-300-9211 Good will, like a good name, is got by many actions, and lost by one. - Francis Jeffrey
Re: Level3 routing issues?
Has someone reported the details to CERT yet? Preferably someone who's got logs and such? -george william herbert [EMAIL PROTECTED]
this attack is still strong here..
[EMAIL PROTECTED] show firewall filter proactive-filter
NameBytes Packets
mssql-drops 916252204 2267951
term NO-MSSQL {
from {
packet-length 404;
protocol udp;
destination-port 1434;
}
then {
count mssql-drops;
discard;
}
}
Re: New worm / port 1434?
On Sat, Jan 25, 2003 at 12:12:37AM -0800, Mike Leber wrote: We are seeing this too. We are seeing the gige interfaces on multiple customer aggregation switches at multiple locations add several hundred Mbps each. All the traffic is destined for udp port 1434 with a randomized source address. We are doing ip verify unicast source reachable-via any which stops most of the random addresses. We've temporarily had to block udp port 1434. USD10 to the first person who spots a CNN reporter speculating to Saddam's involvement.
Re: New worm / port 1434?
1434 is the SQL Server Resolution Service. Unfortunately, this appears to be a whole new thing, I was unable to find anything more recent then May of 2002 about security issues with this port. Thanks, Adam Debus Network Administrator, ReachONE Internet [EMAIL PROTECTED] - Original Message - From: Mike Tancsa [EMAIL PROTECTED] To: Avleen Vig [EMAIL PROTECTED] Cc: [EMAIL PROTECTED] Sent: Friday, January 24, 2003 11:19 PM Subject: Re: New worm / port 1434? Yes, I am seeing this big time. Are you sure its SQL server ? Thats normally 1433 no ? Are there any other details somewhere about this ? At 10:32 PM 1/24/2003 -0800, Avleen Vig wrote: It seems we have a new worm hitting Microsoft SQL server servers on port 1434. Mike Tancsa,tel +1 519 651 3400 Sentex Communications, [EMAIL PROTECTED] Providing Internet since 1994www.sentex.net Cambridge, Ontario Canada www.sentex.net/mike
Re: New worm / port 1434?
From: Mike Tancsa Yes, I am seeing this big time. Are you sure its SQL server ? Thats normally 1433 no ? Are there any other details somewhere about this ? snip All MS SQL servers listen to 1434 reguardless of the other ports they listen on. Depending on configuration depends on what other ports it uses (due to various security models), but 1434 is a constant in all configurations according to a quick search and a read on the last MS SQL vulnerability found in 7/2002. Jack Bates BrightNet Oklahoma
Re: Level3 routing issues?
Appears to relate to this cert advisory http://www.cert.org/advisories/CA-1996-01.html We have it totally blocked on our network but the routers are working over time just rejecting packets. The only way to stop it is to stop MySQL or kill the hosts network connection. [EMAIL PROTECTED] wrote: It is global. 01:42:04.040462 194.87.13.21.1812 x.x.x.x.1434: rad-account-req 376 [id 1] Attr[ User User User User User User User User User User User User User User User User User User User User User User User User User User User User User User User User User [|radius] That is the traffic... On Sat, 25 Jan 2003, hc wrote: I am on Verizon-GNI via Qwest and Genuity and seeing the same problem as well. -hc Joel Perez wrote: I am also seeing increased traffic on my network. It has gotten so bad for one of my edge routers that i cant telnet into it. But i am on Qwest and GBLX. -Original Message- From: Alex Rubenstein [mailto:[EMAIL PROTECTED]] Sent: Sat 1/25/2003 1:04 AM To: hc Cc: [EMAIL PROTECTED] Subject: Re: Level3 routing issues? I dunno about that. But, I am seeing, in the last couple hours, all kinds of new traffic. like, customers who never get attacked or anything, all of a sudden: http://mrtg.nac.net/switch9.oct.nac.net/3865/switch9.oct.nac.net-3865.html We are seeing this on ports all across out network -- nearly 1/2 our ports are in delta alarm right now. Anyone else? I will dig more to look at the traffic. On Sat, 25 Jan 2003, hc wrote: Anyone seeing routing problems with Level3 at this hour? I just witnessed tons of prefixes behind level3's network withdraw. Any information on what is happening (if you know) would be great. Thanks! -hc -- Alex Rubenstein, AR97, K2AHR, [EMAIL PROTECTED], latency, Al Reuben -- --Net Access Corporation, 800-NET-ME-36, http://www.nac.net -- -- Message scanned for viruses and dangerous content by http://www.newnet.co.uk/av/ and believed to be clean
Re: New worm / port 1434?
We had to go through each VLAN to determine which boxes were compromised, looks like W2K SQL. This thing is spreading fast. -D 0. Pete Ashdown [EMAIL PROTECTED] farted: * Avleen Vig ([EMAIL PROTECTED]) [030124 23:50] writeth: It seems we have a new worm hitting Microsoft SQL server servers on port 1434. Affirmative. Be sure to block 1434 UDP on both the inbound and the outbound. Infected servers are VERY NOISY. -- -- http://www.zeromemory.com - metal for your ears.
Re: New worm / port 1434?
I'm seeing obscene amounts of 1434/udp traffic at my transit and peering points. I've filtered it out in both directions everywhere my network touches the outside world. It's almost 20% of my traffic at this point. I think I've calmed the internal storm so far, but we'll see. I saw refence to an ICMP trigger packet. Is there any info on this and is it possible to filter for it w/o killing all ICMP traffic? It'd be nice to know I won't have any more routers or switches fall over tonight. Colo customers seem to be the worst off, the rate limiting kills the router or the traffic kills the backbone. decisions, decisions... -S -- Scott Call Router Geek, ATGi, home of $6.95 Prime Rib Nothing is less productive than to make more efficient what should not be done at all. -Peter Drucker
Re: New worm / port 1434?
Note, further analysis makes me believe that the ICMP we saw immediately
beforehand was a coincidence and unrelated. The origin of the ICMP has
been traced to a customer application.
-jr
* Josh Richards [EMAIL PROTECTED] [20030125 00:21]:
A preliminary look at some of our NetFlow data shows a suspect ICMP payload
delivered to one of our downstream colo customer boxes followed by a
70 Mbit/s burst from them. The burst consisted of traffic to seemingly random
destinations on 1434/udp. This customer typically does about 0.250 Mbit/s
so this was a bit out of their profile. :-) Needless to say, we shut them
down per a suspected security incident. The ICMP came from 66.214.194.31
though that could quite easily be forged or just another compromised box.
We're seeing red to many networks all over the world though our network seems
to have quieted down a bit. Sounds like a DDoS in the works.
Anyone else able to corroborate/compare notes?
Josh Richards jrichard@{ geekresearch.com, cubicle.net, digitalwest.net }
Geek Research, LLC - Digital West Networks, Inc - San Luis Obispo, CA
KG6CYK - IP/Unix/telecom/knowledge/coffee/security/crypto/business/geek
Re: DOS?
On Sat, 25 Jan 2003, Doug Barton wrote: Anyone want to get involved in some sort of real time chat (like IRC) to disuss strategies? We're seeing some pretty big traffic, and related problems in multiple colo's world wide. What's to discuss? If you put something like access-list 150 deny udp any any eq 1434 log-input access-list 150 permit ip any any on all your customer-facing ports you get to 1. filter out the disruptive traffic 2. see which customer systems are infected This works well even on relatively underpowered Cisco 7200 boxes.
Tracing where it started
Hello, It might be interesting if some people were to post when they received their first attack packet, and where it came from, if they happened to be logging. Here is the first packet we logged: Jan 25 00:29:37 EST 216.66.11.120 --Phil ISPrime
Re: New worm / port 1434?
On Sat, Jan 25, 2003 at 08:05:33AM +, Gary Coates wrote: Duplicated info.. But this is an old worm ;-( http://www.cert.org/advisories/CA-1996-01.html This is not the worm that's spreading now. Greetz, Peter -- [EMAIL PROTECTED] | http://www.dataloss.nl/ | Undernet:#clue
Re: dos of the week? was RE: Level3 routing issues?
my transit traffic doubled (luckily it is the low time of the night for me) from 10-12ish I work at a really large east coast University. Our sensors show the problem starting between 12:30-12:45am this morning... Eric :)
Re: DOS?
Hi Any ranges I find I'll echo back to the list. not sure if you've received any nanog mail yet. don't worry about source ip's, unless you're doing to deny '0.0.0.0'. block anything with a destination of udp 1434, find hosts pushing extreme amounts of traffic, get them patched (http://www.microsoft.com/technet/treeview/default.asp?url=/technet/security/bulletin/MS02-039.asp) and then wait for the rest of the internet to catch up... --Rob
Re: nanog list delayed again
On Sat, 25 Jan 2003, Mikael Abrahamsson wrote: Does it really have to be this time everytime something happens and it actually would be nice to get the information out quickly? In this case there may be a causal relationship between the two. Being a mailing list server can't be a fun job when half the net is congested...
Re: New worm / port 1434?
On Sat, 25 Jan 2003, Avleen Vig wrote: On Sat, Jan 25, 2003 at 12:12:37AM -0800, Mike Leber wrote: We are seeing this too. We are seeing the gige interfaces on multiple customer aggregation switches at multiple locations add several hundred Mbps each. All the traffic is destined for udp port 1434 with a randomized source address. We are doing ip verify unicast source reachable-via any which stops most of the random addresses. We've temporarily had to block udp port 1434. USD10 to the first person who spots a CNN reporter speculating to Saddam's involvement. I didnt realise he was such a computer expert!
Re: Level3 routing issues?
On Sat, Jan 25, 2003 at 01:13:30AM -0800, Bill Woodcock wrote: On Sat, 25 Jan 2003, Mikael Abrahamsson wrote: Lots of traffic on udp port 1434 coming in here via TW Telecom and Sprint Looks like we may have a winner for DDoS of the year (so far) What kind of traffic levels are you seeing? I'm working on it for some friends, and I'm seeing about 900mbits/second on a gigabit link coming out of their hosting facility. Lots and lots of Microsoft crap in there, I guess. Somebody remind me why Microsoft is still allowed to exist? Let's not blame MS for admins who don't know how to secure their boxes :-) A patch was released mid-2002 and was also part of SQL Server SP3
Re: Level3 routing issues?
On Sat, 25 Jan 2003, Stephen J. Wilcox wrote: Somebody remind me why Microsoft is still allowed to exist? Dunno, arent they negligent? In any other industry a fundemental flaw would be met with lawsuits, in the computer world tho people seem to get around for some reason. Steve Including the developers of SSHD, HTTPD, NAMED, CVS? How about Linus? Wanna call him up? I am no windows cheerleader, but to think this is something that happens only in windows-land is whack -- might as well put your head in the sand. Simple philosophy: Everything sucks at all times and all places. Routers, switches, hosts, OS's. We, as operators, have to do our best to deal. It's arguable you are as liable as anyone else, since this particular exploit is 'old news' and a patch has been available for it for some time. Also; everyone who just posted to this list made it abundantly clear that they don't have a firewall in front of at least one MS SQL server on their network. Should you really have port 1433/4 open to the world? Would you do this with a MySql server? -- Alex Rubenstein, AR97, K2AHR, [EMAIL PROTECTED], latency, Al Reuben -- --Net Access Corporation, 800-NET-ME-36, http://www.nac.net --
Re: Level3 routing issues?
On Sat, Jan 25, 2003 at 02:57:16AM -0500, Alex Rubenstein wrote: MS SQL, or SQL Monitor? Are those two separate programs? I don't know; I'm not a windows guy. I just watched over the shoulders of a few other techs as they shut what appeared to be everything-MSSQL down. I just found the blinkenlights that were causing the problems, shut those lights off, and pointed the windows guys to the offending boxes :) On Sat, 25 Jan 2003, Blaine Kahle wrote: On Sat, Jan 25, 2003 at 02:05:42AM -0500, Kevin Welch wrote: I am seeing similar traffic loads on my network at this hour, one of our MS SQL servers seemed to be sending a large amount of traffic out to the Internet. Still looking into it but too similar for me to avoid sending an e-mail. Same symptoms here. After disabling MS SQL, which required a reboot as the process didn't want to shut down normally, the traffic stopped. I found 3 boxes on our network that were generating massive amounts of traffic, all of which run MS SQL. -- Blaine Kahle [EMAIL PROTECTED] 0x178AA0E0 -- Alex Rubenstein, AR97, K2AHR, [EMAIL PROTECTED], latency, Al Reuben -- --Net Access Corporation, 800-NET-ME-36, http://www.nac.net -- -- Blaine Kahle [EMAIL PROTECTED] 0x178AA0E0
Re: Level3 routing issues?
On Sat, 25 Jan 2003, Avleen Vig wrote: [snip] Let's not blame MS for admins who don't know how to secure their boxes :-) A patch was released mid-2002 and was also part of SQL Server SP3 Would it not also be a good idea/practice *not* to ever let a MS SQL server (or *any* database server) sit on a network that is directly accessible from the internet ? Having a firewall(s) in front of your database server regardless of the type is pretty much common sense, right? Its bad enough to be stuck having to run/support IIS and MSSQL in any scenario, but letting MSSQL talk to the world just seems like asking for even more trouble. -jon -- + Jon Larsen; Chief Technology Officer, Richweb.com + GnuPG Public Key http://richweb.com/jlarsen.gpg + Richweb.com: Providing Internet-Based Business Solutions since 1995 + Business Telephone: (804) 359.2220 + Jon Larsen Cell Phone: (804) 307.6939
Re: New worm / port 1434?
http://lists.netsys.com/pipermail/full-disclosure/2003-January/003718.html
Re: New worm / port 1434?
From: Eric Gauthier Woot! We made the front page of CNN.com: Electronic attack slows Internet http://www.cnn.com/2003/TECH/internet/01/25/internet.attack/index.html Guess that USD10 goes to some unnamed reporter at CNN And please tell me how CodeRed was worse? I'm sorry, this just created a lot of Internet traffic hurting performance? That's a little underrated. But then again, it's a port that could be blocked and not cause severe damage. Block tcp/80 and people would through a fit. *mental note: Block port 80 anytime another port must be blocked just to be sure. Jack Bates Network Engineer BrightNet Oklahoma
Re: FW: Worm / UDP1434
On Sat, 25 Jan 2003, Freedman David wrote: Anybody here on list using Extreme products (Summit/Alpine/Blackdiamond)? We extensively use extreme networks products in our core, distribution and access. The roadrunner chipset units (Summit24/48) (used mainly for access) dies if you try to put more than say 5 megabit/s of this flood thru it. A lot of purely route-cache products does this, I've talked to people with the same experience with Enterasys units etc. We had a few of those killed off by customers infected and buying 10 megabit/s from us. On the other hand, our inferno chipset units (BDs with MSM64i, Summit48i etc) with EW 6.2.2b56 code handled this just fine. One unit which was directly connected to the customer which tried to put 10 megabit/s of flood thru it complained with some errors but there was never any problems logging into the unit, checking to see where the traffic was from etc. I was able to disable the customers vlan from the customer port and everything went back to normal. Extreme are apparently assembling an advisory TAC on this, from our point of view, since we use the devices to do l3 aggregation (for colo and such) we've used an ACL to try and combat the offending traffic, but its not doing much good. I just did: create access-list block1434 udp destination any ip-port 1434 source any ip-port any deny ports any Bingo, dropping several kpps of traffic thru the switch (BD with MSM64i) hands down, no problemo. I am quite happy with how the I-chipset boxen handled the situation, since they are also route cache based I feared they would really get struck badly but I have seen no such problems. -- Mikael Abrahamssonemail: [EMAIL PROTECTED]
Re: Level3 routing issues?
From what I have read and researched, it does. On Sat, 25 Jan 2003, Jack Bates wrote: From: Avleen Vig snip Let's not blame MS for admins who don't know how to secure their boxes :-) A patch was released mid-2002 and was also part of SQL Server SP3 Has it been verified that the mid-2002/SP3 patches work? I haven't heard anything difinitive on this yet. Jack Bates Network Engineer BrightNet Oklahoma -- Alex Rubenstein, AR97, K2AHR, [EMAIL PROTECTED], latency, Al Reuben -- --Net Access Corporation, 800-NET-ME-36, http://www.nac.net --
Re: Worm / UDP1434
Anybody here on list using Extreme products (Summit/Alpine/Blackdiamond)? They sure don't like this traffic one bit. It causes them to not only drop traffic, but spew out every available error message under the sun... Extreme are apparently assembling an advisory TAC on this, from our point of view, since we use the devices to do l3 aggregation (for colo and such) we've used an ACL to try and combat the offending traffic, but its not doing much good. Do you have MCAST enabled on these switches? I'd guess this is what is causing issues on the extreme boxes. -- Neil J. McRae - Alive and Kicking [EMAIL PROTECTED]
Re: Level3 routing issues?
BIll, - Original Message - From: Bill Woodcock [EMAIL PROTECTED] I'd agree with it. Except the herds of losers who still buy exploding crap from Vendor M don't seem to be thinning themselves out quickly dude, the Exploding Cars are so much easier to drive than the ones from Vendor L. (tic) enough. Maybe they're sexually attractive to each other, and reproduce before their stupidity kills them. That would be unfortunate. Or maybe it's just that none of this computer stuff actually matters, so exploding crap isn't actually fatal. Maybe that's it. I think it sucks that they are exploding on MY highway. With that in mind is it time yet to talk about solutions to problems like this from the network point of view? Sure its easy to put up access list's when needed but I have 100megs available to me on egress and I was trying to push 450megs. Is there anything protocol, vendor specific or otherwise that will not allow rogue machines to at will take up 100% of available resources? I know extreme networks has the concept of Max Port utilization on thier switches, will this help? Suggestions? -Scotty
Re: Level3 routing issues?
On Sat, Jan 25, 2003 at 12:20:41PM -0500, C. Jon Larsen wrote: On Sat, 25 Jan 2003, Avleen Vig wrote: [snip] Let's not blame MS for admins who don't know how to secure their boxes :-) A patch was released mid-2002 and was also part of SQL Server SP3 Would it not also be a good idea/practice *not* to ever let a MS SQL server (or *any* database server) sit on a network that is directly accessible from the internet ? Having a firewall(s) in front of your database server regardless of the type is pretty much common sense, right? Its bad enough to be stuck having to run/support IIS and MSSQL in any scenario, but letting MSSQL talk to the world just seems like asking for even more trouble. I agree absolutely. This is just bad practice and the network admins here need to re-think their security architecture.
Re: Worm / UDP1434
David, - Original Message - From: Freedman David [EMAIL PROTECTED] Anybody here on list using Extreme products (Summit/Alpine/Blackdiamond)? They sure don't like this traffic one bit. It causes them to not only drop traffic, but spew out every available error message under the sun... We use extremes in our core and it did not log much other than CPU issues: 01/25/2003 02:20.23 INFO:SYST task tNetTask cpu utilization is 88% PC: 80266eb4 01/25/2003 02:20.23 CRIT:SYST task tNetTask cpu utilization is 88% PC: 80266eb4 and... 01/25/2003 02:24.43 INFO:SYST task tNetTask cpu utilization is 93% PC: 80266eb4 01/25/2003 02:24.42 CRIT:SYST task tNetTask cpu utilization is 93% PC: 80266eb4 I did notice console messages while investigating the sources of the traffic, but of course have no log of them now. The switches stayed up the whole time though (yay) Also picked up some strange messages from one of the offenders: 01/25/2003 02:23.48 WARN:IPRT IGMP: snooping.c 376: updateGroupSenderListPortMask: PTAGalloc 237.189.185.65/64.237.99.79 01/25/2003 02:23.48 WARN:IPRT IGMP: snooping.c 376: updateGroupSenderListPortMask: PTAGalloc 237.137.210.243/64.237.99.79 01/25/2003 02:23.48 WARN:IPRT IGMP: snooping.c 376: updateGroupSenderListPortMask: PTAGalloc 225.134.14.67/64.237.99.79 No idea yet what that is, though I assume it is coming from the monitor port. -Scotty
Re: UDP 1432
Another data point - I get connectivity through sonic.net (Santa Rosa). This vanished between Fri Jan 24 21:30:00 PST 2003 and Fri Jan 24 21:35:00 PST 2003. At that time, connectivity on other circuits through ALTER.NET, megapath.net and mfnx.net were still ok. All circuits seem to be up now (for us). Fortunately, we don't have any MSQL servers. -- -=[L]=-
Re: Level3 routing issues?
Would it not also be a good idea/practice *not* to ever let a MS SQL server (or *any* database server) sit on a network that is directly accessible from the internet ? Having a firewall(s) in front of your database server regardless of the type is pretty much common sense, right? Its bad enough to be stuck having to run/support IIS and MSSQL in any scenario, but letting MSSQL talk to the world just seems like asking for even more trouble. That depends on what you are using the server for - it might be used by various offices around the world, or to interface with other corporations platforms etc. Ideally this would be in a secured VPN or at the very least be limited by IP address, but MS SQL admins are not alone in the pretend everything will be ok from a security standpoint. Neil. -- Neil J. McRae - Alive and Kicking [EMAIL PROTECTED]
Re: Tracing where it started
On Sat, Jan 25, 2003 at 06:58:46AM -0500, Phil Rosenthal wrote: It might be interesting if some people were to post when they received their first attack packet, and where it came from, if they happened to be logging. Here is the first packet we logged: Jan 25 00:29:37 EST 216.66.11.120 Interestingly, looking through my logs for UDP 1434, I saw a sequential scan of my subnet like so: Jan 16 08:15:51 206.176.210.74,53 - x.x.x.1,1434 PR udp len 20 33 IN Jan 16 08:15:51 206.176.210.74,53 - x.x.x.2,1434 PR udp len 20 33 IN Jan 16 08:15:51 206.176.210.74,53 - x.x.x.3,1434 PR udp len 20 33 IN All from 206.176.210.74, all source port 53 (probably trying to use people's DNS firewall rules to get around being filtered). After that, I saw nothing until the storm started last night from many different source IPs, which was at Jan 24 21:31:53 PST for me. -c
RE: New worm / port 1434?
Codered was worse by the sheer number of hosts that were infected and in the end having a lot more impact than what the SQL Sapphire worm has shown. Now that is not to say this worm does not surpass CodeRed... however it still has its work cut out for it. Last I heard the number of infections ranges from 40k to 200k depending on who you ask. Now if its 200k thats definitely getting close to a CodeRed level however even then it has another few hundred thousand infections to go. The flooding aspect of this worm (it tries to re-infect so fast), it DOES NOT have a ddos engine built into it as some people have mislead, is interesting and is causing a lot of problems for networks. However, its also its downfall as it saturates bandwidth to the point of even it not being able to spread anymore. I could go into other technical details if you like... like how codered properly handled its data manipulation on the stack so that it could keep running whereas Sapphire is going to end up crapping out on itself anyways... and also it does not keep any sort of global flag to thwart off re-infection, therefore once again hindering its ability to spread whereas codered did keep a global atom allowing it to last longer, and infect more. and bla bla bla. You can read both of eEye's analysis of CodeRed and Sapphire here: CodeRed: http://www.eeye.com/html/Research/Advisories/AL20010717.html Sapphire: http://www.eeye.com/html/Research/Flash/AL20030125.html First after soda then after liquor... damn alcoholics. Signed, Marc Maiffret Chief Hacking Officer eEye Digital Security T.949.349.9062 F.949.349.9538 http://eEye.com/Retina - Network Security Scanner http://eEye.com/Iris - Network Traffic Analyzer http://eEye.com/SecureIIS - Stop known and unknown IIS vulnerabilities P.S. Jack and Eric you might be the only ones to get this as I was having trouble earlier posting to NANOG... feel free to forward if you think it matters. | -Original Message- | From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]On Behalf Of | Jack Bates | Sent: Saturday, January 25, 2003 9:36 AM | To: Eric Gauthier; [EMAIL PROTECTED] | Subject: Re: New worm / port 1434? | | | | From: Eric Gauthier | | Woot! | | We made the front page of CNN.com: | | Electronic attack slows Internet | http://www.cnn.com/2003/TECH/internet/01/25/internet.attack/index.html | | Guess that USD10 goes to some unnamed reporter at CNN | | And please tell me how CodeRed was worse? I'm sorry, this just | created a lot | of Internet traffic hurting performance? That's a little underrated. But | then again, it's a port that could be blocked and not cause severe damage. | Block tcp/80 and people would through a fit. | | *mental note: Block port 80 anytime another port must be blocked | just to be | sure. | | Jack Bates | Network Engineer | BrightNet Oklahoma | |
Re: Level3 routing issues?
On Sat, 25 Jan 2003, Alex Rubenstein wrote: Including the developers of SSHD, HTTPD, NAMED, CVS? How about Linus? Wanna call him up? I am no windows cheerleader, but to think this is something that happens only in windows-land is whack -- might as well put your head in the sand. It is interesting to note that one inadvertent advantage of open source (when it requires people to compile from source, and pick and choose options at compile time... popular distributions with precompiled packages obviously break this to a certain degree) is that it leads to a much more heterogenous set of software WRT attacks like buffer overflows. Contrast this to something that is compiled once (or a small handfull) of times by the vendor, resulting in a much more predictable environment for many types of exploits. There have been several worms that have demonstrated this difference. [...] Also; everyone who just posted to this list made it abundantly clear that they don't have a firewall in front of at least one MS SQL server on their network. Should you really have port 1433/4 open to the world? Would you do this with a MySql server? It is interesting to note that apparently Windows NT and 2000 systems default to a somewhat dated and limited ephemeral port range of 1024-5000 (cf. ms kb article 196271). If you are blocking traffic on a variety of inbound UDP ports in that range using a simple packet filter, you will randomly be blocking responses to legitimate outbound UDP traffic, such as DNS. Granted, in many environments there is no need to allow MS systems to directly make DNS queries to anything outside the firewall. There are quite disturbing reports of hosts such as activex.microsoft.com, lawsqlsrv2.hotmail.com, etc. sourcing these packets (ie. appearing to be infected), but they need to be taken with a grain of salt. It is certainly possible that places who have hosts that are otherwise firewalled (that's ok, don't need to patch them...) aren't properly filtering UDP since it is harder to do properly if you require support for UDP traffic.
OK, this is rich
http://www.cnn.com/TECH/ Main story: Electronic attack hits Net A fast-moving computer worm slowed down Internet access Saturday for about 22,000 servers, according to the Internet security firm Symantec. Oliver Friedrichs, a senior manager with Symantec, said the SQL worm was taking advantage of a vulnerability detected six months ago in Microsoft sequel servers, used mainly by companies to store information. Then, first tombstone: MORE NEWS Gates pledges better software security HAHAHAH (props to troy corbin for this) -- Alex Rubenstein, AR97, K2AHR, [EMAIL PROTECTED], latency, Al Reuben -- --Net Access Corporation, 800-NET-ME-36, http://www.nac.net --
Re: Level3 routing issues?
At 11:56 AM 1/25/2003, Bill Woodcock wrote: Dunno, arent they negligent? In any other industry a fundemental flaw would be met with lawsuits, in the computer world tho people seem to get around for some reason. Not true, look at cars and recalls. Also as I understand it MS issued a fix for this sometime ago - it the users who didn't implement it! Uh, lemme see if I get your argument. People who buy exploding cars from Vendor M are at fault when the cars explode, since cars from Vendor M always explode, and Vendor M always disclaims responsibility, since someone usually points out in advance that the cars will explode? To further torture analogies: So what type of vehicles ARE safe for the road, and for which roads? Taking a lawn tractor out on the Interstate surely is the fault of the driver, and not the manufacturer. At what point do folks figure out that putting production servers out on the Internet with no protection whatsoever is an invitation to abuse? Firewalls may not be perfect. Server software may not be perfect. Layering security can sure help. It appears this worm only sought to annoy. Perhaps the next one that goes after the mass of unpatched MS SQL servers will instead take the opportunity to raid these servers for personal information? The opportunities for mass-scale identity theft are rather staggering.
Re: New worm / port 1434?
http://securityresponse.symantec.com/avcenter/venc/data/w32.sqlexp.wor m.html - Original Message - From: Simon Lockhart [EMAIL PROTECTED] To: Mike Tancsa [EMAIL PROTECTED] Cc: Avleen Vig [EMAIL PROTECTED]; [EMAIL PROTECTED] Sent: Saturday, January 25, 2003 3:48 AM Subject: Re: New worm / port 1434? On Sat Jan 25, 2003 at 02:19:04AM -0500, Mike Tancsa wrote: Yes, I am seeing this big time. Are you sure its SQL server ? Thats normally 1433 no ? Are there any other details somewhere about this ? This URL seems to explain the exploit: http://www.nextgenss.com/advisories/mssql-udp.txt Simon -- Simon Lockhart | Tel: +44 (0)1628 407720 (BBC ext 37720) Technology Manager | Fax: +44 (0)1628 407701 (BBC ext 37701) BBC Internet Services | Email: [EMAIL PROTECTED] BBC Technology, Maiden House, Vanwall Road, Maidenhead. SL6 4UB. UK
Re: Level3 routing issues?
Not sure you can claim something you have for free is liable or with guarantee Thats total rubbish. Whether you pay for it or not shouldn't matter. You might also want to consider reading the various software agreement licenses that come with various pieces of software both free and non-free. True altho it does appear to affect MS more so than it ought to even considering their market lead. What evidence do you have here? If I count the number of DDOS attacks from insecure Linux boxes that we've seen in the last year, I'd say that its on par. I expect my purchases to live up to their sales description Yes, thats bad.. people should be more clueful than they are, I blame folks being cheap, having staff who are clueless, low quality equipment, this is the market we're in. Do you actually use MS SQL? From what you've posted I'd say not. Have you had a network outage that your customers have had to suffer? You are blaming yourself in the last statement as its upto operators to make sure customers get the message about securing their network. I've been whining at router manufacturers about alot of their default options for years. Last week I whined at Cisco to put a huge sticker on every CPE router they sell warning about Network Security and Day to Day administration. How much of this to you talk to your own customers about? Or do you just take the money? I don't know of an industry where costs aren't always being lowered. Regards, Neil. -- Neil J. McRae - Alive and Kicking [EMAIL PROTECTED]
FW: FYI - Cisco - Status as of Sat Jan 25...Global worm attack seems related to SQL 2000...see below for patches from Microsoft (available as of 7/17/02).]
-Original Message- From: [mailto:@cisco.com] Sent: Saturday, January 25, 2003 2:13 PM To: Recipient list suppressed Subject: FYI - Cisco - Status as of Sat Jan 25...Global worm attack seems related to SQL 2000...see below for patches from Microsoft (available as of 7/17/02). FYI - According to this article from the Associated Press: http://story.news.yahoo.com/news?tmpl=story2ncid=716e=3u=/ap/2003012 5/ap_on_hi_te/internet_attack http://story.news.yahoo.com/news?tmpl=story2ncid=716e=3u=/ap/20030125 /ap_on_hi_te/internet_attack The attack sought to exploit a software flaw discovered by researchers in July 2002 that permits hackers to seize control of corporate database servers. Microsoft deemed the flaw to be critical and offered a free repairing patch, but it was impossible to know how many computer administrators applied the fix. Symptoms that may be seen, detected and may be causing alerts on Cisco devices include, but are not limited to high CPU and traffic drops on the input interfaces. The Microsoft Security advisory specifies that this vulnerability is specific to SQL 2000. Microsoft first published the fixed patch on 7/17/2002. Please insure that you are at the correct patch levels for all your servers that use SQL 2000. Microsoft Security Bulletin MS02-039 http://www.microsoft.com/technet/treeview/default.asp?url=/technet/secur ity/bulletin/MS02-039.asp This is basically the same attack as code red using the same UDP port numbers to for the attack. If you have applied patches for the code red virus they are most likely covered protected. The attached link from CNN does a nice job of explaining the similarity. http://www.cnn.com/2003/TECH/internet/01/25/internet.attack.ap/index.htm l Cisco utilizes a security harden OS for servers running our services such Call Manager 3.3. Though SQL 2000 is used by Cisco Unity and Call Manager 3.3, it is still appropriate and best practice to keep all servers current with the latest patches to avoid known vulnerabilities and protect against future re-occurrences. Cisco's Host Intrusion Detection System (HIDS) can be used on servers to detect unknown attacks, as was Code Red prior to patches being available. Thanks, Cisco == TECHNICAL INFORMATION - There is a Global attack going on around the world which is a WORM that is attacking the Microsoft SQL server on UDP port 1433 1434. Cisco TAC has the following PSIRT that can be used to help our Customer. * Summary: Cisco customers may currently be experiencing attacks due to a new worm that has hit the Internet. The signature of this worm appears to be high volumes of UDP traffic to port 1434. Affected customers have been experiencing high volumes of traffic from both internal and external systems. Symptoms that can be seen detected on Cisco devices include, but are not limited to high CPU and traffic drops on the input interfaces. Details: UDP port 1433 and 1434 are used for SQL server traffic. A new worm has been targeting this port and attempting to exploit a buffer overflow vulnerability in Microsoft's SQL server. Microsoft has issued a security advisory about this issue, the details are here: http://www.microsoft.com/technet/treeview/default.asp?url=/technet/secur ity/bulletin/MS02-039.asp For infected servers, MS recommends downloading Service Pack 3 for SqlSvr, located here: http://www.microsoft.com/sql/downloads/2000/sp3.asp?SD=GN http://www.microsoft.com/sql/downloads/2000/sp3.asp?SD=GNLN=en-usgssn b=1 LN=en-usgssnb=1 Symptoms: You may see instability in networks due to increased load. The traffic load generated by this DoS is very high, with some customers experiencing traffic loads as high as 20 Megabits per second combined egress and ingress rates. Workarounds until patches can be applied: Thus far the best mitigation is to block inbound and outbound traffic destined to UDP port 1434. Care must be taken in regards to the impact on mission critical services as 1434/udp and 1433/udp are used by Microsoft SQL Server. Before blocking traffic to that port completely make sure that the possible effects on your network are understood. PLEASE NOTE: These workarounds block both ports 1433 and 1434, although we have received no evidence yet that blocking port 1433 has any affect on the attack. If your network requires traffic to flow on port 1433 please leave that portion of the ACL out and monitor your results closely. VACL config on 5500/6500 - confirmed that this drops the CPU load on the MSFC as well. set security acl ip WORM deny udp any eq 1433 any set security acl ip WORM deny udp any any eq 1433 set security acl ip WORM deny udp any any eq 1434 set security acl ip WORM deny udp any eq 1434 any set security acl ip WORM permit any commit security acl WORM set
Re: Level3 routing issues?
I think you are on the right lines below in suggesting that products and services should be supplied safe and not require additional maintenance out of the box to make them so (additional changes should make them weaker) There is no such thing as safe! You have control over what risks you want to take the aim should always be to lower them but if you want safe, pull the power plug, place your box in a large metal container and sink it in very deep waters. I don't know of an industry where costs aren't always being lowered. I dont know of one where prices are below cost values such that players of all sizes regularly go bankrupt and services are squeezed harder and harder. Microsoft and XBox is an example, lots of industries have loss leaders. Still waiting on evidence that most security issues are due to Microsoft though! Regards, Neil. -- Neil J. McRae - Alive and Kicking [EMAIL PROTECTED]
Re: Level3 routing issues?
On 1/25/03 2:53 PM, Christopher L. Morrow [EMAIL PROTECTED] wrote: Keep in mind that these problems aren't from 'well behaved' hosts, and 'well behaved' hosts normally listen to ECN/tcp-window/Red/WRED classic DoS attack scenario. :( Well not everyone plays fair out there. I imagine this is built into SLA's too right? My network will be up as long as everyone is well behaved I understand the evils, but are we really at the mercy of situations like this? Of course we can firewall the common sense things ahead of time, and we can jump right in and block evil traffic when it happens, after it takes down our network but what sorts of things can we design into our networks today to help with these situations? -Scotty
Re: Level3 routing issues?
On Saturday 25 January 2003 10:03 am, Avleen Vig wrote: On Sat, Jan 25, 2003 at 12:20:41PM -0500, C. Jon Larsen wrote: On Sat, 25 Jan 2003, Avleen Vig wrote: [snip] Let's not blame MS for admins who don't know how to secure their boxes :-) A patch was released mid-2002 and was also part of SQL Server SP3 Would it not also be a good idea/practice *not* to ever let a MS SQL server (or *any* database server) sit on a network that is directly accessible from the internet ? Having a firewall(s) in front of your database server regardless of the type is pretty much common sense, right? Its bad enough to be stuck having to run/support IIS and MSSQL in any scenario, but letting MSSQL talk to the world just seems like asking for even more trouble. I agree absolutely. This is just bad practice and the network admins here need to re-think their security architecture. Sometimes that's just not an option. We operate a colo facility, and while we strongly encourage best practices customers don't always listen. My personal firewall will protect me etc... It's just unfortunate when one person's ignorance leads to problems for other people, as in this case. -- Grant A. Kirkwood - grant(at)tnarg.org Fingerprint = D337 48C4 4D00 232D 3444 1D5D 27F6 055A BF0C 4AED
Re: DOS?
On Sat, 25 Jan 2003, Iljitsch van Beijnum wrote: On Sat, 25 Jan 2003, Rob Thomas wrote: ] access-list 150 deny udp any any eq 1434 log-input Be _very_ careful about enabling such logging. Some of the worm flows have filled GigE pipes. I doubt you really want to log that; Netflow is a better option in this case. Too much logging will raise the CPU utilization to the point of creating a DoS on the router. As a general rule, yes. But: Access list logging does not show every packet that matches an entry. Logging is rate-limited to avoid CPU overload. What logging shows you is a reasonably representative sample, but not a complete packet trace. Remember that there are packets you're not seeing. either way, the logging for this, ESPECIALLY with log-input, is a dangerous proposition. One thing to keep in mind is that the S-train platforms are different in handling logging than the normal trains... so S-train rate-limits (and bumps out them annoying messages about rate-limited messages) while others punt as much to the route processor as possible and happily saturate it :( (Don't log on like a 7500 for instance if the packet rates are over like 5kpps...) Access lists and logging have a performance impact, but not a large one. Be careful on routers running at more than about 80 percent CPU load, or when applying access lists to very high-speed interfaces. right, or on platforms not built to scale :) (like 7500 or smaller boxen) ( http://www.cisco.com/warp/public/707/22.html ) There doesn't seem to be a noticable impact on CPU usage for a C12000 GigE linecard. Can you do Netflow rather than CEF on such a beast without a performance penalty? One thing to keep in mind is that perhaps you don't care about the logging :) Just drop it and make your customers fix their borked boxes...
Re: W32.SqlSlammer
Drew, There *IS* a difference between windows SP3 and Microsoft SQL2000 SP3.. you do know that right? -Scotty By the way, I know you guys probably don't care but McAfee is saying that if you have SP3 on your windows2000 server you will not be infected with SQLSlammer, this is absolutely NOT true, I have a box with sp3 and it IS infected. -Drew
Re: Level3 routing issues?
Third point to the correlation above: The vast majority of Windows admins are dingbat-morons, self-proclaimed experts. Had then not been dingbat-morons, and applied the readily available and widely announced patches (as zealously as unix folks patch thier stuff), this'd be all moot, and we'd all have gotten a better nights sleep. I don't think this is fair statement either, Linux and Microsoft have the most issues because they have the largest market share - security by obscurity. It doesn't mean they have anymore issues than any other vendors, success brings problems and this is one of them. Regards, Neil. -- Neil J. McRae - Alive and Kicking [EMAIL PROTECTED]
Re: Level3 routing issues?
On Sat, 25 Jan 2003, Neil J. McRae wrote: I think you are on the right lines below in suggesting that products and services should be supplied safe and not require additional maintenance out of the box to make them so (additional changes should make them weaker) There is no such thing as safe! You have control over what risks you want to take the aim should always be to lower them but if you want safe, pull the power plug, place your box in a large metal container and sink it in very deep waters. Agreed but on the assumption people will connect their new PC to the Internet the supplied OS should be appropriately configured. I don't know of an industry where costs aren't always being lowered. I dont know of one where prices are below cost values such that players of all sizes regularly go bankrupt and services are squeezed harder and harder. Microsoft and XBox is an example, lots of industries have loss leaders. Still waiting on evidence that most security issues are due to Microsoft though! A loss leader does not cause bankruptcy, they have a profitable section to sustain the loss making product. In our industry we just seem to run with too small a margin. Hmm dont think I can argue the Linux vs MS point tho, its a big can of worms! This may be academic tho in our discussion, are you saying COLT uses MS servers in favour of linux for its public services? The question of which is more secure depends on numbers, application, etc I see loads of linux patches every month that I dont install because I have not installed or disabled most features in my OS. I believe if you count security bulletins linux has in fact overtaken microsoft. On the other hand if you count incidents you'll find the Codered, Nimda and probably this one too at the top of the list. But then offset that against the market penetration MS has into joe public.. and so on. Heres my advice to the uninitiated. Run linux, run firewalls, disable what you dont need and listen to folks who have real world experience. Steve
Re: Level3 routing issues?
On Sat, 25 Jan 2003, K. Scott Bethke wrote: BIll, - Original Message - From: Bill Woodcock [EMAIL PROTECTED] I'd agree with it. Except the herds of losers who still buy exploding crap from Vendor M don't seem to be thinning themselves out quickly dude, the Exploding Cars are so much easier to drive than the ones from Vendor L. (tic) unfortunately (being a vendor L user myself) you must admit that these too have problems :( (at times) enough. Maybe they're sexually attractive to each other, and reproduce before their stupidity kills them. That would be unfortunate. Or maybe it's just that none of this computer stuff actually matters, so exploding crap isn't actually fatal. Maybe that's it. I think it sucks that they are exploding on MY highway. With that in mind is it time yet to talk about solutions to problems like this from the network point of view? Sure its easy to put up access list's when needed but I have 100megs available to me on egress and I was trying to push 450megs. Is there anything protocol, vendor specific or otherwise that will not allow rogue machines to at will take up 100% of available resources? I know extreme networks has the concept of Max Port utilization on thier switches, will this help? Suggestions? Keep in mind that these problems aren't from 'well behaved' hosts, and 'well behaved' hosts normally listen to ECN/tcp-window/Red/WRED classic DoS attack scenario. :(
Re: Level3 routing issues?
On Sat, 25 Jan 2003, Avleen Vig wrote: On Sat, Jan 25, 2003 at 12:20:41PM -0500, C. Jon Larsen wrote: On Sat, 25 Jan 2003, Avleen Vig wrote: [snip] Let's not blame MS for admins who don't know how to secure their boxes :-) A patch was released mid-2002 and was also part of SQL Server SP3 Would it not also be a good idea/practice *not* to ever let a MS SQL server (or *any* database server) sit on a network that is directly accessible from the internet ? Having a firewall(s) in front of your database server regardless of the type is pretty much common sense, right? Its bad enough to be stuck having to run/support IIS and MSSQL in any scenario, but letting MSSQL talk to the world just seems like asking for even more trouble. I agree absolutely. This is just bad practice and the network admins here need to re-think their security architecture. I've not looked at any great detail into the exact sources but of the few I looked at earlier I was surprised to find them on ADSL .. these may be corporate networks this is the bit I dont know but some of them seemed to be residential, weird! Steve
Re: Level3 routing issues?
On Sat, Jan 25, 2003 at 05:08:22PM +, Stephen J. Wilcox wrote: Also; everyone who just posted to this list made it abundantly clear that they don't have a firewall in front of at least one MS SQL server on their network. Should you really have port 1433/4 open to the world? Would you do this with a MySql server? Yes, thats bad.. people should be more clueful than they are, I blame folks being cheap, having staff who are clueless, low quality equipment, this is the market we're in. The market we are in was specifically bred by Microsoft in the 90's when they claimed Windows was so eay to use, anyone could admin it. They've since changed their tune, but the damage has been done and continues to be done like last night :(
Re: Level3 routing issues?
What about doing some priority-based QoS? If a single IP exceeds X amount of traffic, prioritize traffic above that threshold as low. It would keep any one single host from saturating a link if the threshold is low. For example, you may say that each IP is limited to 10mb of prioirty traffic. Yes, a compromised host may try to barf out 90mb of chaff, but the excess would be moved down the totem pole. Obviously, this may not make sense in all environments, but in a campus or large enterprise situation, I can see this occuring on your WAN links in particular. On Sat, 25 Jan 2003, K. Scott Bethke wrote: BIll, - Original Message - From: Bill Woodcock [EMAIL PROTECTED] I'd agree with it. Except the herds of losers who still buy exploding crap from Vendor M don't seem to be thinning themselves out quickly dude, the Exploding Cars are so much easier to drive than the ones from Vendor L. (tic) enough. Maybe they're sexually attractive to each other, and reproduce before their stupidity kills them. That would be unfortunate. Or maybe it's just that none of this computer stuff actually matters, so exploding crap isn't actually fatal. Maybe that's it. I think it sucks that they are exploding on MY highway. With that in mind is it time yet to talk about solutions to problems like this from the network point of view? Sure its easy to put up access list's when needed but I have 100megs available to me on egress and I was trying to push 450megs. Is there anything protocol, vendor specific or otherwise that will not allow rogue machines to at will take up 100% of available resources? I know extreme networks has the concept of Max Port utilization on thier switches, will this help? Suggestions? -Scotty
Re: W32.SqlSlammer
At 02:21 PM 1/25/2003, you wrote: By the way, I know you guys probably don't care but McAfee is saying that if you have SP3 on your windows2000 server you will not be infected with SQLSlammer, this is absolutely NOT true, I have a box with sp3 and it IS infected. To clarify, we're talking about Microsoft SQL Server 2000, Service Pack 3, not the Windows 2000 Service Pack 3 (which also exists). Two completely different animals I've got one machine with SQL 2K on it, and it, too, was infected. Then I installed SQL Server 2000 SP3, and put it back on the net. Just to be sure, I opened up port 1434 to it, and sat back and watched. Lotta traffic to port 1434, but nothing happened. It got hit several times, and never joined the crowd spewing traffic. If you have an infected machine, pull it off the 'net... immediately, if not sooner. Then go download the service pack for the SQL Server at: http://www.microsoft.com/downloads/details.aspx?FamilyId=9032F608-160A-4537-A2B6-4CB265B80766displaylang=en Getting the 44 meg file over to the disconnected server is left as an exercise for the reader (remember SneakerNet?)
Re: 1434 traffic
What I'm seeing from on my personal network connections is a lot of traffic to udp port 1434 start at 05:30:08 UTC. I did some graphing of reports we got to DShield/ISC up to 9am EST. http://isc.sans.org/port1434start.gif The part that amazes me is the speed. It saturated within 1 minute! Does anybody else see the oscillations in traffic? I remember seeing something similar in netflow data for slapper (2002 udp). Or is this just an artifact of our particular dataset? So far, we got about 80,000 sources (distinct IPs sending port 1434 packets) -- [EMAIL PROTECTED] Collaborative Intrusion Detection join http://www.dshield.org
Re: W32.SqlSlammer
On Sat, Jan 25, 2003 at 02:21:21PM -0500, Drew Weaver wrote: By the way, I know you guys probably don't care but McAfee is saying that if you have SP3 on your windows2000 server you will not be infected with SQLSlammer, this is absolutely NOT true, I have a box with sp3 and it IS infected. They probably meant MS SQL Service Pack 3.
Re: Level3 routing issues?
On Sat, 25 Jan 2003, Stephen J. Wilcox wrote: I've not looked at any great detail into the exact sources but of the few I looked at earlier I was surprised to find them on ADSL .. these may be corporate networks this is the bit I dont know but some of them seemed to be residential, weird! Seems this borked software bit is in more than just hardcore SQLServer. It seems that the bits are also in visio2000 and a few other things :( Hence the 'more than server platform' infection spread. This also helps to explain the speed of infection and spread, as with more possible targets things should move more quickly. The interesting is the huge spike at a common time (00:30EST) one wonders if there is a group tracking down the initial infector or not :)
Does the Worm have another Payload besides 1434 Floods?
So the worm is sending out tons of UDP1434 packets that let it break into MS-SQL servers and reproduce, and that's certainly annoying because of the traffic floods. But is it carrying anything else that will do more damage, or anything that leaves it a security hole to be exploited later? It would be really annoying if machines that aren't cleaned up later reformat themselves or hang out waiting for further instructions. Also, several people have commented that restarting their MS-SQL servers stops the problem. Does it just stop the flooding, but leave code there, or does the worm strictly live in transitory data space that's really gone after a restart. Several people have talked about bursts of ICMP or 6667 traffic, and those are probably unrelated, but maybe not. (What? More than one cracker on the net or more than one program that chokes when overloaded? Who'd'a' thunk it!)
Re: Tracing where it started
* Clayton Fiske ([EMAIL PROTECTED]) [030125 12:55] writeth: On Sat, Jan 25, 2003 at 06:58:46AM -0500, Phil Rosenthal wrote: It might be interesting if some people were to post when they received their first attack packet, and where it came from, if they happened to be logging. Here is the first packet we logged: Jan 25 00:29:37 EST 216.66.11.120 Interestingly, looking through my logs for UDP 1434, I saw a sequential scan of my subnet like so: Jan 16 08:15:51 206.176.210.74,53 - x.x.x.1,1434 PR udp len 20 33 IN I'm not sure that going back that far is going to offer anything conclusive, as it could have been any number of scanners looking for vulnerabilities. Looking at my logs back to the 19th, I have isolated hits on the 19th and 23rd. However, they really started to come in force at 22:29:39 MDT, two seconds after Clayton's. My first attempt came from an IP owned by Level 3 Comm. Jan 23 02:43:44 c6509-core 10829487: 47w0d: %SEC-6-IPACCESSLOGP: list 130 denied udp 192.41.65.170(48962) - 166.70.10.63(1434), 1 packet Jan 24 22:29:39 c6509-core 10966964: 47w1d: %SEC-6-IPACCESSLOGP: list 130 denied udp 65.57.250.28(1210) - 204.228.150.9(1434), 1 packet Jan 24 22:29:44 border 7577864: 30w2d: %SEC-6-IPACCESSLOGP: list 100 denied udp 129.219.122.204(1170) - 204.228.132.100(1434), 1 packet Jan 24 22:29:50 border 7577865: 30w2d: %SEC-6-IPACCESSLOGP: list 100 denied udp 212.67.198.3(1035) - 166.70.22.47(1434), 1 packet Jan 24 22:29:52 xmission-paix 425068: 7w0d: %SEC-6-IPACCESSLOGP: list 100 denied udp 61.103.121.140(3546) - 166.70.22.87(1434), 1 packet Jan 24 22:29:52 border 7577868: 30w2d: %SEC-6-IPACCESSLOGP: list 100 denied udp 65.57.250.28(1210) - 204.228.132.18(1434), 1 packet Jan 24 22:29:55 c6509-core 10966977: 47w1d: %SEC-6-IPACCESSLOGP: list 130 denied udp 61.103.121.140(3546) - 166.70.10.8(1434), 1 packet Jan 24 22:29:57 c6509-core 10966979: 47w1d: %SEC-6-IPACCESSLOGP: list 130 denied udp 12.24.139.231(3315) - 204.228.140.81(1434), 1 packet Jan 24 22:29:58 c6509-core 10966980: 47w1d: %SEC-6-IPACCESSLOGP: list 130 denied udp 140.115.113.252(3780) - 207.135.133.228(1434), 1 packet Jan 24 22:29:59 c6509-core 10966981: 47w1d: %SEC-6-IPACCESSLOGP: list 130 denied udp 17.193.12.215(3117) - 207.135.155.209(1434), 1 packet Jan 24 22:30:00 border 7577873: 30w2d: %SEC-6-IPACCESSLOGP: list 100 denied udp 209.15.147.225(4543) - 204.228.133.186(1434), 1 packet
Re: W32.SqlSlammer
On Sat Jan 25, 2003 at 02:21:21PM -0500, Drew Weaver wrote: By the way, I know you guys probably don't care but McAfee is saying that if you have SP3 on your windows2000 server you will not be infected with SQLSlammer, this is absolutely NOT true, I have a box with sp3 and it IS infected. I think you'll find it's SP3 for SQLServer you need, not SP3 for Win2k. Simon -- Simon Lockhart | Tel: +44 (0)1628 407720 (BBC ext 37720) Technology Manager | Fax: +44 (0)1628 407701 (BBC ext 37701) BBC Internet Services | Email: [EMAIL PROTECTED] BBC Technology, Maiden House, Vanwall Road, Maidenhead. SL6 4UB. UK
Re: Tracing where it started
It might be interesting if some people were to post when they received their first attack packet, and where it came from, if they happened to be logging. Here is the first packet we logged: Jan 25 00:29:37 EST 216.66.11.120 A quick followup to my previous message. I found an earlier attempt in the *:29 window on my home firewall. I don't know if this is due to Cisco logging lag or what. In any case, its interesting how relatively close it is to Phil's IP, but they are different networks. Again the time is in MDT: Jan 24 22:29:25 chariot kernel: fp=UDP-FORWARD:1 a=DROP IN=eth0 OUT=eth3 SRC=216.64.162.15 DST=166.70.201.243 LEN=404 TOS=0x00 PREC=0x00 TTL=111 ID=4917 PROTO=UDP SPT=2958 DPT=1434 LEN=384
How to find the first occurrance of the worm.
Ray Burkholder -Original Message- From: McDonald, Dan [mailto:[EMAIL PROTECTED]] Sent: January 25, 2003 17:05 To: '[EMAIL PROTECTED]' Subject: [flow-tools] w32.sqlexp.worm In case anyone needs it, here is the flow-tools nfilter that I've found to match the worm that hit us... filter-primitive mssql type ip-port permit 1434 default deny filter-primitive wormsize type counter permit eq 404 default deny filter theworm match src-ip-port mssql match octets wormsize that with a flow-print -f 5 gave me the time of the first infection... Daniel J McDonald, CCIE #2495, CNX Lan/Wan Integrator Austin Energy 1.512.322.6739 [EMAIL PROTECTED] ___ [EMAIL PROTECTED] http://www.splintered.net/sw/flow-tools
worm design (Re: Level3 routing issues?)
MS Date: Sat, 25 Jan 2003 10:17:01 -0800 (PST) MS From: Marc Slemko MS It is interesting to note that one inadvertent advantage of open MS source (when it requires people to compile from source, and pick MS and choose options at compile time... popular distributions with MS precompiled packages obviously break this to a certain degree) is MS that it leads to a much more heterogenous set of software WRT MS attacks like buffer overflows. 1. Position-relative opcodes used in shellcode 2. Syscalls triggered via a software trap, not subroutine call 3. Dynamic linking involves fixups stored in the binary 4. Activate a syscall, then check the stack to find %eip Eddy -- Brotsman Dreger, Inc. - EverQuick Internet Division Bandwidth, consulting, e-commerce, hosting, and network building Phone: +1 (785) 865-5885 Lawrence and [inter]national Phone: +1 (316) 794-8922 Wichita ~ Date: Mon, 21 May 2001 11:23:58 + (GMT) From: A Trap [EMAIL PROTECTED] To: [EMAIL PROTECTED] Subject: Please ignore this portion of my mail signature. These last few lines are a trap for address-harvesting spambots. Do NOT send mail to [EMAIL PROTECTED], or you are likely to be blocked.
Re: Level3 routing issues?
## On 2003-01-25 20:04 - Stephen J. Wilcox typed: SJW SJW SJW Heres my advice to the uninitiated. Run linux, run firewalls, disable what you SJW dont need and listen to folks who have real world experience. SJW SJW Steve SJW Please don't start a flame war about this but are you implying that the Major Linux distributions are the most secure Unix-like OS (at least out of the box) ??? -- Thanks Rafi
Re: Level3 routing issues?
On Sun, 26 Jan 2003, Rafi Sadowsky wrote: ## On 2003-01-25 20:04 - Stephen J. Wilcox typed: SJW SJW SJW Heres my advice to the uninitiated. Run linux, run firewalls, disable what you SJW dont need and listen to folks who have real world experience. SJW SJW Steve SJW Please don't start a flame war about this but are you implying that the Major Linux distributions are the most secure Unix-like OS (at least out of the box) ??? I hadnt really thought about it, I was just offering my approach to running servers on the public Internet Dont read too much into it, I wasnt suggesting that snippet as the absolute way to connect to the internet.. it was preceded by a discussion on where folks place their database servers.. Steve
Banc of America Article
http://biz.yahoo.com/rb/030125/tech_virus_boa_1.html Let's make the assumption that the outage of ATM's that BoA suffered was caused by last nights 'SQL Slammer' virus. The following things can then be assumed: a) BoA's network has Microsoft SQL Servers on them. b) BoA has not applied SP3 (available for a week) or the patch for this particular problem (SQL Slammer) (available for many months). c) somehow, this attack spawned on the public internet made it's way to BoA's SQL servers, bypassing firewalls (did they have firewalls?). Another article states, Bank of America Corp., one of the nation's largest banks, said many customers could not withdraw money from its 13,000 ATM machines because of technical problems caused by the attack. A spokeswoman, Lisa Gagnon, said the bank restored service to nearly all ATMs by late Saturday afternoon and that customers' money and personal information had not been at risk. Does anyone else, based upon the assumptions above, believe this statement to be patently incorrect (specifically, the part about 'personal information had not been at risk.') ? I find these statement made by BoA, based upon assumptions which are probably correct, to be very scary. Comments? -- Alex Rubenstein, AR97, K2AHR, [EMAIL PROTECTED], latency, Al Reuben -- --Net Access Corporation, 800-NET-ME-36, http://www.nac.net --
Re: Tracing where it started
According to Clayton Fiske: Interestingly, looking through my logs for UDP 1434, I saw a sequential scan of my subnet like so: Jan 16 08:15:51 206.176.210.74,53 - x.x.x.1,1434 PR udp len 20 33 IN Jan 16 08:15:51 206.176.210.74,53 - x.x.x.2,1434 PR udp len 20 33 IN Jan 16 08:15:51 206.176.210.74,53 - x.x.x.3,1434 PR udp len 20 33 IN All from 206.176.210.74, all source port 53 (probably trying to use people's DNS firewall rules to get around being filtered). After that, I saw nothing until the storm started last night from many different source IPs, which was at Jan 24 21:31:53 PST for me. Ditto on the sequential scan well before the actual action, except that mine came on Jan. 19th: Jan 19 10:59:11 Deny inbound UDP from 67.8.33.179/1 to xxx.xxx.xxx.xxx ... ... The scan went across several subnets I manage inside 209.67.0.0 serially. My sources were all from 67.8.33.179, all source port 1. The actual worm propagation began to hit my logs at 00:28:16 EST Jan 25. Cheers. -travis
Re: OK, this is rich
And don't forget to check for a conspicuously absent article on the front page of www.msn.com. On Sat, Jan 25, 2003 at 01:56:41PM -0500, Alex Rubenstein eloquently stated: http://www.cnn.com/TECH/ Main story: Electronic attack hits Net A fast-moving computer worm slowed down Internet access Saturday for about 22,000 servers, according to the Internet security firm Symantec. Oliver Friedrichs, a senior manager with Symantec, said the SQL worm was taking advantage of a vulnerability detected six months ago in Microsoft sequel servers, used mainly by companies to store information. Then, first tombstone: MORE NEWS Gates pledges better software security HAHAHAH (props to troy corbin for this) -- Alex Rubenstein, AR97, K2AHR, [EMAIL PROTECTED], latency, Al Reuben -- --Net Access Corporation, 800-NET-ME-36, http://www.nac.net -- -- Stephen Milton - Vice President(425) 881-8769 x102 ISOMEDIA.COM - Premium Internet Services(425) 869-9437 Fax [EMAIL PROTECTED]http://www.isomedia.com
Re: DOS?
On Sat, 25 Jan 2003, Christopher L. Morrow wrote: wants to log for a while and then counts hits against the cache until it only for identical packets... so source A:123 - Dest B:80 x50 packets gets logged 'once'. One log for the first packet and update logs at 5 min intervals (which may be setable in some ios command, which may only exist in S-train code). If the attack is randomized, sources, destinations, or ports... there is effecively a new 'flow' for each packet and thus a new log message for each... (again, in S-train code or 12.0(21)+ code this is rate-limited to the RP and thus to the logs... somewhat atleast) It seems the flow recognition isn't that strict but I might just have been lucky. actually logs. This should work very well, and it does as per my tests on a heavily loaded 4500 router. So why would one type of IOS do this right and another version that isn't immediately recognizable by the version number as inferior do it wrong? S-train code has specific features that don't get propogated to other trains because they aren't 'required' there or aren't applicable, or not asked for. Lovely when others decide what you require. I think today's events show that CPU-based routers have no business handling anything more than 1 x 100 Mbps in and 1 x 100 Mbps out. If a box has 40 FE interfaces or 4 GE interfaces, at some point you'll see 4 Gbps coming in so the box must be able to handle it to some usable degree. that may be, but CPE isn't normally vendor J for t1/t3/oc3 customers... CPE for T1 would be 2500, T3 3600, OC3 7200 or some such. All are fine for day-to-day stuff but don't pack enough power to handle today's events at line rate. But the difference is small enough that it can be remedied by simply using faster CPUs. Those were available at the time the boxes were introduced, but I assume a faster CPU would have increased the cost price too much. never mind dsl/dial/cable customers, eh? Those are slow enough to be done in software easily. The vast majority is cpu based equipment. Whether or not that's a good thing is immaterial, no one is going to upgrade all ruouting gear overnight :( (or in 2 years as we've seen) People are buying GE equipment left right and center too. It doesn't make much sense to have more computing power in the ethernet chip (GE over UTP takes a lot of processing power) than in the chip doing the routing. Maybe its possible to find some middle ground, for instance by doing some basic flow recognition and rate limiting in hardware but the actual routing in software. That way, you can build a GE CPE router that can do 100 kpps which is enough for regular traffic but still have some protection when there is a 1.4 Mpps DoS attack which would otherwise have killed the CPU. That's why I want the logging: to see which customer is spewing out the garbage. (-: well, then.. log vs log-input :) cause log-input is more processing and thus more pain. (and if its 'inbound' on interfaces the 'log-input' is kinda pointless, eh? Good point. The reason it's there is that I didn't know what I was dealing with when I enabled this logging and I wanted to see the MAC addresses in case the source IP addresses were spoofed.
Re: Tracing where it started
Our first (this is EST): Jan 25 00:29:44 external.firewall1.oct.nac.net firewalld[109]: deny in eth0 404 udp 20 114 61.103.121.140 66.246.x.x 3546 14 34 (default) 61.103.121.140 = a host somewhere on GBLX On Sat, 25 Jan 2003, Pete Ashdown wrote: * Clayton Fiske ([EMAIL PROTECTED]) [030125 12:55] writeth: On Sat, Jan 25, 2003 at 06:58:46AM -0500, Phil Rosenthal wrote: It might be interesting if some people were to post when they received their first attack packet, and where it came from, if they happened to be logging. Here is the first packet we logged: Jan 25 00:29:37 EST 216.66.11.120 Interestingly, looking through my logs for UDP 1434, I saw a sequential scan of my subnet like so: Jan 16 08:15:51 206.176.210.74,53 - x.x.x.1,1434 PR udp len 20 33 IN I'm not sure that going back that far is going to offer anything conclusive, as it could have been any number of scanners looking for vulnerabilities. Looking at my logs back to the 19th, I have isolated hits on the 19th and 23rd. However, they really started to come in force at 22:29:39 MDT, two seconds after Clayton's. My first attempt came from an IP owned by Level 3 Comm. Jan 23 02:43:44 c6509-core 10829487: 47w0d: %SEC-6-IPACCESSLOGP: list 130 denied udp 192.41.65.170(48962) - 166.70.10.63(1434), 1 packet Jan 24 22:29:39 c6509-core 10966964: 47w1d: %SEC-6-IPACCESSLOGP: list 130 denied udp 65.57.250.28(1210) - 204.228.150.9(1434), 1 packet Jan 24 22:29:44 border 7577864: 30w2d: %SEC-6-IPACCESSLOGP: list 100 denied udp 129.219.122.204(1170) - 204.228.132.100(1434), 1 packet Jan 24 22:29:50 border 7577865: 30w2d: %SEC-6-IPACCESSLOGP: list 100 denied udp 212.67.198.3(1035) - 166.70.22.47(1434), 1 packet Jan 24 22:29:52 xmission-paix 425068: 7w0d: %SEC-6-IPACCESSLOGP: list 100 denied udp 61.103.121.140(3546) - 166.70.22.87(1434), 1 packet Jan 24 22:29:52 border 7577868: 30w2d: %SEC-6-IPACCESSLOGP: list 100 denied udp 65.57.250.28(1210) - 204.228.132.18(1434), 1 packet Jan 24 22:29:55 c6509-core 10966977: 47w1d: %SEC-6-IPACCESSLOGP: list 130 denied udp 61.103.121.140(3546) - 166.70.10.8(1434), 1 packet Jan 24 22:29:57 c6509-core 10966979: 47w1d: %SEC-6-IPACCESSLOGP: list 130 denied udp 12.24.139.231(3315) - 204.228.140.81(1434), 1 packet Jan 24 22:29:58 c6509-core 10966980: 47w1d: %SEC-6-IPACCESSLOGP: list 130 denied udp 140.115.113.252(3780) - 207.135.133.228(1434), 1 packet Jan 24 22:29:59 c6509-core 10966981: 47w1d: %SEC-6-IPACCESSLOGP: list 130 denied udp 17.193.12.215(3117) - 207.135.155.209(1434), 1 packet Jan 24 22:30:00 border 7577873: 30w2d: %SEC-6-IPACCESSLOGP: list 100 denied udp 209.15.147.225(4543) - 204.228.133.186(1434), 1 packet -- Alex Rubenstein, AR97, K2AHR, [EMAIL PROTECTED], latency, Al Reuben -- --Net Access Corporation, 800-NET-ME-36, http://www.nac.net --
Re: Does the Worm have another Payload besides 1434 Floods?
From: Stewart, William C (Bill), SALES But is it carrying anything else that will do more damage, or anything that leaves it a security hole to be exploited later? It would be really annoying if machines that aren't cleaned up later reformat themselves or hang out waiting for further instructions. All disassembly analasis made shows that it is a simplistic worm designed to break in, execute, and start sending itself out. No system damage or host embedding has been detected. The writer of the worm had no intentions of causing permanent damage. Also, several people have commented that restarting their MS-SQL servers stops the problem. Does it just stop the flooding, but leave code there, or does the worm strictly live in transitory data space that's really gone after a restart. It's really gone after a restart. Several people have talked about bursts of ICMP or 6667 traffic, and those are probably unrelated, but maybe not. (What? More than one cracker on the net or more than one program that chokes when overloaded? Who'd'a' thunk it!) Paranoia. Engineers ignore a lot of things until something critical hits. Then they go overboard analyzing every little packet that doesn't seem right. In general, as most EUs are finding out as they install them pesky firewalls, the 'net is full of noise. Jack Bates Network Engineer BrightNet Oklahoma
Re: Level3 routing issues?
From: Robert A. Hayden What about doing some priority-based QoS? If a single IP exceeds X amount of traffic, prioritize traffic above that threshold as low. It would keep any one single host from saturating a link if the threshold is low. For example, you may say that each IP is limited to 10mb of prioirty traffic. Yes, a compromised host may try to barf out 90mb of chaff, but the excess would be moved down the totem pole. snip Down the totem pole isn't off the totem pole. In most cases the issue wasn't traffic priority. Most network equipment isn't designed to handle 100% capacity from all ports. Under standard operation, maximum capacity is never reached. It is cost prohibitive to support it. In addition, this was a dual issue. Not only did the bandwidth saturate, the packets are so small that in reaching for 100% saturation, many routers and switches first exceeded their maximum pps thresholds. The best defense is to monitor and know your traffic. When traffic becomes uncommon, someone needs to be alerted. A 30% processor increase is not a good thing; ever. Second, know the optimizations for your particular equipment and code. Each piece of equipment has it's own optimizations. In my case, it was better to access-list at the router level than to run bandwidth limiting, and I run a crummy 7200. It's even nicer on a 7500+ where it's offloaded to the linecard processors. If a portion of the network or a specific port is unrecoverable, shut it down. The server won't be able to handle traffic anyways, and it is better to cut off a portion of the network than lose the entire network. Jack Bates Network Engineer BrightNet Oklahoma
Re: Banc of America Article
From: Alex Rubenstein Does anyone else, based upon the assumptions above, believe this statement to be patently incorrect (specifically, the part about 'personal information had not been at risk.') ? Actually, the statements are correct. Remember, the worm wasn't programmed to put the database or the security of the networks at risk. Of course, the customer's information could have been at risk, but in hind sight, it wasn't. However, there is another possibility. BofA could have piped a portion of the public network through equipment that sustains their private network in a secure manner. However, a MS-SQL system (or a couple hundred) which contained nothing of value was infected. The load created by the system was enough to interrupt equipment along the path and effectively shut down their private network even though it didn't have direct access. Example, I can run IP through ATM switches. The overloading of the PVC could systematically destroy the integrity of the ATM network which holds other ATM traffic. This is still a secure model, although obviously not well engineered as proper ATM CoS would have limited the IP traffic. Of course, ATM would be one example. They could be tunneling IP over any number of protocols commonly used by banks. In essence, only one piece of common equipment has to be shut down to cause a problem. Jack Bates BrightNet Oklahoma
Re: Level3 routing issues?
On Sat, Jan 25, 2003 at 02:10:59PM -0800, Stephen Milton wrote: We have had multiple customers who had SP3 on their boxes that were hit. SP3 was _supposed_ to include this patch, there is no verification so far that it did. Since all the providers have been blocking the attack spread from the routers, installing SP3 on boxes post-attack hasn't really been put to the test yet. Did you install WIDOWS service pack 3 or SQL SERVER service pack 3?
Re: Level3 routing issues?
MS SQL SP3, _NOT_ MS Windows 2000 SP3. BIG DIFFERENCE. http://www.microsoft.com/sql/downloads/2000/sp3.asp On Sat, 25 Jan 2003, Stephen Milton wrote: We have had multiple customers who had SP3 on their boxes that were hit. SP3 was _supposed_ to include this patch, there is no verification so far that it did. Since all the providers have been blocking the attack spread from the routers, installing SP3 on boxes post-attack hasn't really been put to the test yet. YMMV On Sat, Jan 25, 2003 at 08:40:53AM -0800, Avleen Vig eloquently stated: Let's not blame MS for admins who don't know how to secure their boxes :-) A patch was released mid-2002 and was also part of SQL Server SP3 -- Stephen Milton - Vice President(425) 881-8769 x102 ISOMEDIA.COM - Premium Internet Services(425) 869-9437 Fax [EMAIL PROTECTED]http://www.isomedia.com -- Alex Rubenstein, AR97, K2AHR, [EMAIL PROTECTED], latency, Al Reuben -- --Net Access Corporation, 800-NET-ME-36, http://www.nac.net --
Re: DOS?
On Sat, 25 Jan 2003, Iljitsch van Beijnum wrote: On Sat, 25 Jan 2003, Christopher L. Morrow wrote: Access list logging does not show every packet that matches an entry. Logging is rate-limited to avoid CPU overload. either way, the logging for this, ESPECIALLY with log-input, is a dangerous proposition. Are you saying that I shouldn't believe Cisco's own documentation? Obviously, it's going to take _some_ CPU cycles, but I would expect the box to remain operational. Yes, you'd expect this to remain operational.. but the real world 'testing' shows that not to be the case. If the attack has highly random source or destination the log messages get gen'd for each packet :( This causes a little pain (or alot if you qualify dropping routing protocols as alot) on the router :( CPU spikes due to logging large floods are quite common. This I know from very personal experience. One thing to keep in mind is that the S-train platforms are different in handling logging than the normal trains... Ok, I've been working with Cisco equipment for 8 years now and I can configure them in my sleep, but all the version/image/train/feature set is still voodoo to me. Obviously, the router caches the information it me too. wants to log for a while and then counts hits against the cache until it only for identical packets... so source A:123 - Dest B:80 x50 packets gets logged 'once'. One log for the first packet and update logs at 5 min intervals (which may be setable in some ios command, which may only exist in S-train code). If the attack is randomized, sources, destinations, or ports... there is effecively a new 'flow' for each packet and thus a new log message for each... (again, in S-train code or 12.0(21)+ code this is rate-limited to the RP and thus to the logs... somewhat atleast) actually logs. This should work very well, and it does as per my tests on a heavily loaded 4500 router. So why would one type of IOS do this right and another version that isn't immediately recognizable by the version number as inferior do it wrong? S-train code has specific features that don't get propogated to other trains because they aren't 'required' there or aren't applicable, or not asked for. possible and happily saturate it :( (Don't log on like a 7500 for instance if the packet rates are over like 5kpps...) I think today's events show that CPU-based routers have no business handling anything more than 1 x 100 Mbps in and 1 x 100 Mbps out. If a box has 40 FE interfaces or 4 GE interfaces, at some point you'll see 4 Gbps coming in so the box must be able to handle it to some usable degree. that may be, but CPE isn't normally vendor J for t1/t3/oc3 customers... never mind dsl/dial/cable customers, eh? The vast majority is cpu based equipment. Whether or not that's a good thing is immaterial, no one is going to upgrade all ruouting gear overnight :( (or in 2 years as we've seen) There doesn't seem to be a noticable impact on CPU usage for a C12000 GigE linecard. Can you do Netflow rather than CEF on such a beast without a performance penalty? One thing to keep in mind is that perhaps you don't care about the logging :) Just drop it and make your customers fix their borked boxes... That's why I want the logging: to see which customer is spewing out the garbage. (-: well, then.. log vs log-input :) cause log-input is more processing and thus more pain. (and if its 'inbound' on interfaces the 'log-input' is kinda pointless, eh?
Banc of America Article
http://biz.yahoo.com/rb/030125/tech_virus_boa_1.html Let's make the assumption that the outage of ATM's that BoA suffered was caused by last nights 'SQL Slammer' virus. The following things can then be assumed: a) BoA's network has Microsoft SQL Servers on them. b) BoA has not applied SP3 (available for a week) or the patch for this particular problem (SQL Slammer) (available for many months). c) somehow, this attack spawned on the public internet made it's way to BoA's SQL servers, bypassing firewalls (did they have firewalls?). Another article states, Bank of America Corp., one of the nation's largest banks, said many customers could not withdraw money from its 13,000 ATM machines because of technical problems caused by the attack. A spokeswoman, Lisa Gagnon, said the bank restored service to nearly all ATMs by late Saturday afternoon and that customers' money and personal information had not been at risk. Does anyone else, based upon the assumptions above, believe this statement to be patently incorrect (specifically, the part about 'personal information had not been at risk.') ? I find these statement made by BoA, based upon assumptions which are probably correct, to be very scary. Comments? -- Alex Rubenstein, AR97, K2AHR, [EMAIL PROTECTED], latency, Al Reuben -- --Net Access Corporation, 800-NET-ME-36, http://www.nac.net --
Re: Banc of America Article
On Sat, 25 Jan 2003, Alex Rubenstein wrote: Does anyone else, based upon the assumptions above, believe this statement to be patently incorrect (specifically, the part about 'personal information had not been at risk.') ? Patently incorrect? No. It is possible. Even if the confidentiality of your data is protected, you are still vulnerability to attacks on availability and integrity of the data. For example, you may fully encrypt all your data, use VPNs, etc. But you can still loose service due to network congestion or routers failing due to other unprotected traffic on your network. One of the most common mistakes I see rookie security people make is thinking confidentiality is the only business requirement.
Re: 13,000 Bank of America ATM's taken out by virus.
On Sat, 25 Jan 2003, Christopher J. Wolff wrote: Does this mean that BofA ATM's are SQL based or that BofA is running ATM traffic through some kind of internet VPN? Perhaps they just plug the ATM's into any connection and pass cleartext transactions over the internet? This is very suspicious, IMHO. At $previous_employer half the connections to the various banks they had were via VPN. /\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\ Patrick Greenwell Asking the wrong questions is the leading cause of wrong answers \/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/
Re: Level3 routing issues?
On Sat, Jan 25, 2003 at 08:56:06AM -0800, Bill Woodcock wrote: Dunno, arent they negligent? In any other industry a fundemental flaw would be met with lawsuits, in the computer world tho people seem to get around for some reason. Not true, look at cars and recalls. Also as I understand it MS issued a fix for this sometime ago - it the users who didn't implement it! Uh, lemme see if I get your argument. People who buy exploding cars from Vendor M are at fault when the cars explode, since cars from Vendor M always explode, and Vendor M always disclaims responsibility, since someone usually points out in advance that the cars will explode? I'm not sure that your argument has anything to do with the law or with right and wrong, but in a sort of social-Darwinism sort of way, I guess I'd agree with it. Except the herds of losers who still buy exploding crap from Vendor M don't seem to be thinning themselves out quickly enough. Maybe they're sexually attractive to each other, and reproduce before their stupidity kills them. That would be unfortunate. Or maybe it's just that none of this computer stuff actually matters, so exploding crap isn't actually fatal. Maybe that's it. Time for someone to fight the product liability included in the 'shrinkwrap' licenses. I do believe that there should be some sort of inital grace period for the software industry.. they are well intentioned and not as old as the car industry.. but the dire affects and lost sleep for some people need to eventually be reckoned with. The grace period should probally be over now and the industry declared 'mature and liable' for shoddy software. If my car has a recall notice, i get a letter saying dear sir, your gas tank may explode if used. please come in for our inspection. If they can keep track of those millions of cars each year, at least somewhat, it should be simple to track who purchased the software and send them a letter saying get these patches now.. or perhaps they can do some agrement with AOL to include all the latest patches in those CDs^H^H^HCoasters they send me. - Jared -- Jared Mauch | pgp key available via finger from [EMAIL PROTECTED] clue++; | http://puck.nether.net/~jared/ My statements are only mine.
