I was finally able to track down the infected PC's (yes, more than one). Below is a brief description of what occurred and the fix. First, thanks to all that responded to me.
As previously mentioned, I had an attack on a customer of mines network that was showing up as follows: SrcIf SrcIPaddress DstIf DstIPaddress Pr SrcP DstP Pkts Fa0/1 127.0.0.124 Se1/2.500 108.122.0.0 00 0000 0000 285 The above capture is just 1 of a few hundred packets similar to it and all coming from a different source address on the 127.0.0.0 network. The amount of traffic was so large that at times it peaked to over 20MB and as a result it overran my WAN interfaces causing BGP to flap / reconverge. Just when BGP got a chance to come back up and learned all 115000 routes, the attack occured again and the links would flap. Pingging the 127.0.0.x IP address from the edge router where the attack was initially spotted did not give me any replies. All I got were UUUUU. I was also not able to ping the broadcast address as all it gave me was UUUUU (unreachables) as well. There was no ARP entries on that router for that IP. I ended up enabling Netflow on the edge router (what you see above) in order to get more detail of what was going on. I got to see what interface it was coming in on so I applied an access-list on the router to filter out these packets. That allow the router and bgp to stabilize. The next thing was to move on to the switch that was connected to this FA0/1 interface. This switch has a router module, I ended up doing the same thing as I did on the edge router except this time I also connected to the sc0 interface and I enabled one port as the mirroring port on the switch and placed a PC with Etherreal to monitor everything that was destined to 108.122.0.0 and I finally got a MAC address. I issued the show CAM command on the switch and it told me where it came from which was another switch. I moved on to that other switch. The MAC address that was being reported was the MSM route module of that switch. I enabled netflow on it as well and I was able to see the vlan that the attack was coming on and the VLAN where it was destined to. Luckily there were only 2 PCs (Sun Spark Stations) on that vlan and both were compromised. I removed them from the network and all is well. I did also have MRTG which help some with identifying when the attack was going on and what direction it was coming on and with the ports that were being most heavily utilized. This network is pretty big so it was difficult to monitor all the ports that were suspects. Thank you all again for your help. As far as the runt packets are concerned, to tell you the truth, I noticed that but did not pay to much attention to that part of the Netflow output since I was all wrapped up on tracking down where these packets were coming in from. Right now packets with size of 1-32 account for about 50% of all traffic. Thanks, Mario Puras SoluNet Technical Support Mailto: [EMAIL PROTECTED] Direct: (321) 309-1410 888.449.5766 (USA) / 888.SOLUNET (Canada) -----Original Message----- From: jhodge [mailto:[EMAIL PROTECTED]] Sent: Friday, December 27, 2002 4:34 PM To: [EMAIL PROTECTED] Subject: RE: Possible Attack???? [7:59813] Not sure if this will help, but you could enable ip accounting on the uplink interface to the switch. Watch for the address that is pouring out the most requests. Then use sho ip arp x.x.x.x to find the mac address. From there you could go to the switch and do a show cam dynamic or if IOS version, show mac-address-table with the mac address found with the most requests. This would hunt down the culprit machine without a person walking to each individual machine. Cheers, -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Sam Sneed Sent: December 27, 2002 1:04 PM To: [EMAIL PROTECTED] Subject: Re: Possible Attack???? [7:59813] Do you run SNMP and mrtg on theswitch? You can than graphically see which host has been pouring out all the traffic with ease. wrote in message [EMAIL PROTECTED]">news:[EMAIL PROTECTED]... > Thanks Priscilla. I figure it was some sort of spoofing which is what I > ended up reporting last night. The traffic on the edge router is under > controll. I was able to narrow down which VLAN on the switch it was coming > in on. There is someone going onsite this morning and we are going to work > on narrawing down the actual culprit PC. It should not be difficult to spot > by looking at the LED on the switch (I hope). The attack seems to come in > spurts but when it comes, I see anywhere from about 3000-15000 packets per > second that last about 10 seconds. The weird thing is that when I remove > the access-list that is currently filtering the 127 address, the attack last > much longer. It is almost like it knows that the access-list has been > removed. Since the traffic that I am filtering is not related to ICMP then > I know that I am not sending out any Unreachable message back to the source. > > > > > > Thanks, > > Mario Puras > SoluNet Technical Support > Mailto: [EMAIL PROTECTED] > Direct: (321) 309-1410 > 888.449.5766 (USA) / 888.SOLUNET (Canada) > > > > -----Original Message----- > From: Priscilla Oppenheimer [mailto:[EMAIL PROTECTED]] > Sent: Thursday, December 26, 2002 10:57 PM > To: [EMAIL PROTECTED] > Subject: RE: Possible Attack???? [7:59813] > > > Sending with a source address of 127.x.x.x is often used in IP spoofing. You > should try to find out which station is doing this. It could be compromised. > Of course, it will be hard to find, but if the packets haven't crossed a > router, the MAC address will have a clue. The first six bytes of the MAC > address are a vendor code. Of course, if all your equipment is from one > vendor, that doesn't help much! > > The destination address of 108.122.0.0 is strange also. I looked it up in > the ARIN Whois database and it says it's part of a range reserved by IANA. > I'm not sure why it's reserved, but it seems like a suspicious address to > use. > > So, you're doing the right thing to filter out these packets. > > But you said the problem remained. The other thing I noticed that's strange > is probably unrelated to a possible attack. > > Why are 75% of your packets in the 1-32 byte range? Those are illegal runt > frames on Ethernet. Could you have a duplex mismatch problem?? You should > check the output of show int Fa0/1. > > Good luck! > > Priscilla > > [EMAIL PROTECTED] wrote: > > > > Hi all. I was wondering if someone can share some light on a > > wierd issues > > that I am seeing. This perhaps maybe an attack from an > > internal or infected > > host within the network or simply a malfunctioning NIC. > > Basically, I have a > > Cisco 3662 with 2 Satellite links. I noticed that the main WAN > > link > > (1.544mb) was bursting outbound to sometimes 20mb. I noticed a > > lot of > > output drops and the links started to flap and as a result BGP > > sessions > > starting going down causing huge problems. Once I was able to > > get the BGP > > under control, I enabled Netflow on the inbound interface > > (FE0/1) to see > > what type of traffic could be causing this issue and this is > > when I noticed > > the below: > > > > > > Here is the output of the Netflow: > > > > cisco_3600_one#show ip cache flow > > IP packet size distribution (4096357 total packets): > > 1-32 64 96 128 160 192 224 256 288 320 352 384 > > 416 448 > > 480 > > .753 .167 .017 .005 .001 .002 .001 .001 .001 .001 .000 .000 > > .000 .000 > > .000 > > > > 512 544 576 1024 1536 2048 2560 3072 3584 4096 4608 > > .000 .001 .008 .005 .027 .000 .000 .000 .000 .000 .000 > > > > IP Flow Switching Cache, 278544 bytes > > 978 active, 3118 inactive, 121929 added > > 2503952 ager polls, 0 flow alloc failures > > last clearing of statistics never > > Protocol Total Flows Packets Bytes Packets > > Active(Sec) > > Idle(Sec) > > -------- Flows /Sec /Flow /Pkt /Sec > > /Flow /Flow > > TCP-Telnet 41 0.0 50 40 0.0 > > 31.3 14.4 > > TCP-FTP 87 0.0 7 65 0.0 > > 17.0 12.1 > > TCP-FTPD 27 0.0 135 211 0.0 > > 83.0 3.5 > > TCP-WWW 43121 0.3 8 335 2.8 > > 3.6 2.7 > > TCP-SMTP 1137 0.0 6 173 0.0 > > 9.8 9.7 > > TCP-BGP 1 0.0 673 68 0.0 > > 1796.8 3.6 > > TCP-Frag 2 0.0 1 40 0.0 > > 0.0 15.5 > > TCP-other 33285 0.2 14 246 3.7 > > 24.0 10.3 > > UDP-DNS 6005 0.0 1 73 0.0 > > 1.3 15.4 > > UDP-NTP 10 0.0 1 76 0.0 > > 0.0 15.4 > > UDP-other 13772 0.1 6 78 0.7 > > 1.2 15.5 > > ICMP 2904 0.0 3 72 0.0 > > 19.1 15.4 > > IP-other 20559 0.1 148 20 24.5 > > 6.8 15.4 > > Total: 120951 0.9 33 76 32.2 > > 9.9 9.4 > > > > > > . > > . > > . > > SrcIf SrcIPaddress DstIf DstIPaddress Pr > > SrcP DstP > > Pkts > > Fa0/1 127.0.0.124 Se1/2.500 108.122.0.0 00 > > 0000 0000 > > 285 > > Fa0/1 127.0.0.125 Se1/2.500 108.122.0.0 00 > > 0000 0000 > > 38 > > Fa0/1 127.0.0.122 Se1/2.500 108.122.0.0 00 > > 0000 0000 > > 35 > > Fa0/1 127.0.0.123 Se1/2.500 108.122.0.0 00 > > 0000 0000 > > 296 > > Fa0/1 127.0.0.120 Se1/2.500 108.122.0.0 00 > > 0000 0000 > > 33 > > Fa0/1 127.0.0.121 Se1/2.500 108.122.0.0 00 > > 0000 0000 > > 36 > > Fa0/1 127.0.0.118 Se1/2.500 108.122.0.0 00 > > 0000 0000 > > 52 > > Fa0/1 127.0.0.116 Se1/2.500 108.122.0.0 00 > > 0000 0000 > > 189 > > Fa0/1 127.0.0.117 Se1/2.500 108.122.0.0 00 > > 0000 0000 > > 277 > > Fa0/1 127.0.0.114 Se1/2.500 108.122.0.0 00 > > 0000 0000 > > 32 > > Fa0/1 127.0.0.115 Se1/2.500 108.122.0.0 00 > > 0000 0000 > > 215 > > Fa0/1 127.0.0.112 Se1/2.500 108.122.0.0 00 > > 0000 0000 > > 177 > > Fa0/1 127.0.0.113 Se1/2.500 108.122.0.0 00 > > 0000 0000 > > 80 > > Fa0/1 127.0.0.110 Se1/2.500 108.122.0.0 00 > > 0000 0000 > > 234 > > Fa0/1 127.0.0.111 Se1/2.500 108.122.0.0 00 > > 0000 0000 > > 279 > > Fa0/1 127.0.0.108 Se1/2.500 108.122.0.0 00 > > 0000 0000 > > 171 > > Fa0/1 127.0.0.109 Se1/2.500 108.122.0.0 00 > > 0000 0000 > > 139 > > Fa0/1 127.0.0.106 Se1/2.500 108.122.0.0 00 > > 0000 0000 > > 151 > > Fa0/1 127.0.0.107 Se1/2.500 108.122.0.0 00 > > 0000 0000 > > 57 > > Fa0/1 127.0.0.104 Se1/2.500 108.122.0.0 00 > > 0000 0000 > > 67 > > Fa0/1 127.0.0.105 Se1/2.500 108.122.0.0 00 > > 0000 0000 > > 34 > > Fa0/1 127.0.0.102 Se1/2.500 108.122.0.0 00 > > 0000 0000 > > 272 > > Fa0/1 127.0.0.103 Se1/2.500 108.122.0.0 00 > > 0000 0000 > > 144 > > Fa0/1 127.0.0.100 Se1/2.500 108.122.0.0 00 > > 0000 0000 > > 88 > > . > > . > > . > > . > > > > > > The list goes on and on showing 127.x.x.x. If you notice that > > the incoming > > interface is my Fast Ethernet interface but the incoming source > > address is a > > 127.x.x.x. It is going out my WAN link (the same one that has > > been peaking > > to ~20MB) destined to a boggus Network. The protocol is boggus > > as well. > > > > > > I enabled an access-list to block this 127.x.x.x address > > inbound from my FE > > interface and that has seem to take care of the spikes but the > > problem is > > still present. Anyone have any ideas what this could be????? > > > > > > > > > > Thanks, > > > > Mario Puras > > SoluNet Technical Support > > Mailto: [EMAIL PROTECTED] > > Direct: (321) 309-1410 > > 888.449.5766 (USA) / 888.SOLUNET (Canada) Message Posted at: http://www.groupstudy.com/form/read.php?f=7&i=59885&t=59813 -------------------------------------------------- FAQ, list archives, and subscription info: http://www.groupstudy.com/list/cisco.html Report misconduct and Nondisclosure violations to [EMAIL PROTECTED]
