Re: hey had eric sent you
Bit hard by same bug. What version of code are you running on the 6513 8.1(2) fixes the bug on the 6x48 line cards. What happens is that packets of 64 bytes or less are silently dropped. Replacing linecards will not help unless there is another bug of which I am not aware. With a little digging I can dredge up the relevant DDTS. Scott C. McGrath On Sat, 13 Mar 2004, joe wrote: MessageThis in reply to the earlier thread Weird Problems? Well barring that, I've seen simuliar issues, maybe not the exact same timings but. I've noticed a couple of things while working with a roll out of Active-Directory and a recent upgrade to I.E 6.0 for the user base. Since there were several thousand users involved some of the issues were simply bad configs/drivers/etc. However one of the stats I have noticed is that in certain situations where a system is connected to a Cisco 3548, and the client is running in an Auto detect (AD/AN) mode that things are horendiously slow during boot up, and at various times seem to hang unexplainably. It seemed corrected by setting the client to 100/Full, but not in all cases. Lots of HTTP complaints still remain about accessing webpages etc. but never consistant. This of course is a pretty fresh problem and is still in my queue for research to start this Monday. As well, we've found that there was an odd bug with Cisco's 6513s and their 48 port 10/100/1000 line cards. This was using the latest IOS/CAT software at the time. Again not sure if its a documented problem but, several users were unable to Telnet or FTP to systems that teminated to the 6513, oddly we were able to icmp echo and pass HTTP. After sometime and 2 TACs I found that there was a bug regarding these items and real small packets (I Think less than 64bits??) being passed thru the 6513 and got an RMA for new Line cards. Again, perhaps nothing to do with your situation. Since the Nix systems and non-Doze seem not to have an issue, perhaps you can enlighten me with further Sniffs/Captures of these events directly? As soon as I get some more data/Captures on my end from the problems I'm seeing I can forward those apon request so as to keep S/N ratio down on Nanog (: Cheers, -Joe - Original Message - From: Riley, Marty To: [EMAIL PROTECTED] Sent: Friday, March 12, 2004 11:17 PM Subject: FW: hey had eric sent you
Cisco 6513 Bug (was Re: hey had eric sent you
Scott, Yep, we had to send in the line cards to get them upgraded, didn't have any information on upgrading the s/w on the Line cards and TAC wanted me to RMA them back. So. Boy this one was a real pain because it only seemed protocol specific at the time. Here's the referenced Bug for those interested. CSCeb67650 Bug Details Headline WS-X6548-GE-TX WS-X6148-GE-TX may drop frames on egress Product cat6000 Model x6548 Component hw-1000tx Duplicate of Severity 2 Severity help Status Resolved Status help First Found-in Version 8.1 All affected versions First Fixed-in Version 8.1(1.8), 8.1(1.9), 8.2(0.18)DEL, 7.6(2.3), 12.1(19.4)E, 12.2(17a)SX Version help Release Notes Packets destined out the WS-X6548-GE-TX or the WS-X6148-GE-TX that are less than 64 bytes will be dropped. This can occur when a device forwards a packet that is 60 bytes and the 4 byte dot1q tag is to added to create a valid 64 byte packet. When the tag is removed the packet is 60 bytes. If the destination is out a port on the WS-X6548-GE-TX or the WS-X6148-GE-TX it will be dropped by the linecard. Additionally, if packets are received on an interface that does not have a minimum MTU of 64 bytes (e.g. ATM,POS) and are destined out the WS-X6548-GE-TX or the WS-X6148-GE-TX it will be dropped by the linecard. No current workaround other than moving the recieving device to a different model linecard. Cheers! -Joe -- From: [EMAIL PROTECTED]:[EMAIL PROTECTED] on behalf of Scott McGrath[SMTP:[EMAIL PROTECTED] Sent: Monday, March 15, 2004 11:07 AM To: joe Cc: Riley, Marty; [EMAIL PROTECTED] Subject:Re: hey had eric sent you Bit hard by same bug. What version of code are you running on the 6513 8.1(2) fixes the bug on the 6x48 line cards. What happens is that packets of 64 bytes or less are silently dropped. Replacing linecards will not help unless there is another bug of which I am not aware. With a little digging I can dredge up the relevant DDTS. Scott C. McGrath On Sat, 13 Mar 2004, joe wrote:
FW: hey had eric sent you
Title: Message I'm running short on theories and options, so I thought I would see if anyother ISPshave seen this problem on your network(s). If so, what was the cure? mjr -Original Message- The Unknown problem. Symptoms: At random times dialup, dedicated, internal network users are unable to pass TCP traffic to off network sites. ICMP and UDP appears to be uneffected by the outage which lasts anywhere from 2 to 5 minutes. The problem appears to be wide spread with similar reports from WVNET and other ISPS. nTelos is experiencing a similar problem but we have yet to confirm it is the same. Problem has changed in it's action but remained similar enough to consider it the same problem. Effected Platforms: Windows 2000 Pro, XP Home, XP Pro, 2003 Server. Uneffected Platforms: Unix, MacOS (?) History: During the week of 2/9/04 the call center started recieving reports of users being unable to connect to sites off the CityNet network. Sites hosted on the internal network are uneffected by the outage. Initally it was thought to be a Internet Explorer problem possably caused by the KB832894 / IE SP1 or other updates but after further investigation it was found that Mozilla users were encountering the same problem. After several days of testing it was determined that during the outage any TCP session started on any port would fail. TCP sessions started before the outage continue to work and show no ill effects from the outage. After logging connection attempts at various intervals on many machines there appears to be no sort of pattern in the outages. Most machines encounter the problem, some more than others and a few do not encounter it at all. The duration and frequency of the outage is very fluid. During an outage, we can verify that the packet does seem to leave and reenter the network: Mar 5 22:28:04 pittpa-chaswv-ds3 17083: SLOT 2:6d20h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.14.174(3376) - 216.41.224.3(80), 1 packet Mar 5 22:28:09 pittpa-chaswv-ds3 17084: SLOT 1:6d20h: %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) - 69.43.14.174(3376), 1 packet Mar 5 22:28:09 pittpa-chaswv-ds3 17085: SLOT 2:6d20h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.14.174(3378) - 216.41.224.3(80), 1 packet Mar 5 22:28:09 pittpa-chaswv-ds3 17086: SLOT 1:6d20h: %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) - 69.43.14.174(3378), 1 packet Mar 5 22:33:24 pittpa-chaswv-ds3 17089: SLOT 1:6d20h: %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) - 69.43.14.174(3378), 7 packets Mar 5 22:33:24 pittpa-chaswv-ds3 17090: SLOT 1:6d20h: %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) - 69.43.14.174(3376), 17 packets Mar 5 22:33:58 pittpa-chaswv-ds3 17092: SLOT 2:6d20h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.14.174(3378) - 216.41.224.3(80), 7 packets Mar 5 22:33:58 pittpa-chaswv-ds3 17093: SLOT 2:6d20h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.14.174(3376) - 216.41.224.3(80), 18 packets Mar 5 00:58:30 pittpa-clarwv-ds3 16062: SLOT 2:5d22h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3183) - 216.41.224.3(80), 1 packet Mar 5 00:58:30 pittpa-clarwv-ds3 16063: SLOT 1:5d22h: %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) - 69.43.23.23(3183), 1 packet Mar 5 01:03:28 pittpa-clarwv-ds3 16067: SLOT 2:5d22h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3217) - 216.41.224.3(80), 1 packet Mar 5 01:03:28 pittpa-clarwv-ds3 16068: SLOT 1:5d22h: %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) - 69.43.23.23(3217), 1 packet Mar 5 01:03:34 pittpa-clarwv-ds3 16069: SLOT 2:5d22h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3228) - 216.41.224.3(80), 1 packet Mar 5 01:03:34 pittpa-clarwv-ds3 16070: SLOT 1:5d22h: %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) - 69.43.23.23(3228), 1 packet Mar 5 01:03:39 pittpa-clarwv-ds3 16072: SLOT 2:5d22h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3239) - 216.41.224.3(80), 1 packet Mar 5 01:03:47 pittpa-clarwv-ds3 16073: SLOT 2:5d22h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3183) - 216.41.224.3(80), 74 packets Mar 5 01:04:13 pittpa-clarwv-ds3 16075: SLOT 1:5d22h: %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) - 69.43.23.23(3183), 72 packets Mar 5 01:08:46 pittpa-clarwv-ds3 16078: SLOT 2:5d22h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3218) - 216.41.224.3(80), 4 packets Mar 5 01:08:46 pittpa-clarwv-ds3 16079: SLOT 2:5d22h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3217) - 216.41.224.3(80), 3 packets Mar 5 01:08:46 pittpa-clarwv-ds3 16080: SLOT 2:5d22h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3221) - 216.41.224.3(80), 19 packets Mar 5 01:08:46 pittpa-clarwv-ds3 16081: SLOT 2:5d22h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp
Port 80 Navigation Problem (was:FW: hey had eric sent you)
Title: Message My apologies to the list - I haven't slept in a day or two and completely forgot to make a semi-intelligent subject line... mjr -Original Message-From: Riley, Marty Sent: Friday, March 12, 2004 22:18To: [EMAIL PROTECTED]Subject: FW: hey had eric sent you I'm running short on theories and options, so I thought I would see if anyother ISPshave seen this problem on your network(s). If so, what was the cure? mjr -Original Message- The Unknown problem. Symptoms: At random times dialup, dedicated, internal network users are unable to pass TCP traffic to off network sites. ICMP and UDP appears to be uneffected by the outage which lasts anywhere from 2 to 5 minutes. The problem appears to be wide spread with similar reports from WVNET and other ISPS. nTelos is experiencing a similar problem but we have yet to confirm it is the same. Problem has changed in it's action but remained similar enough to consider it the same problem. Effected Platforms: Windows 2000 Pro, XP Home, XP Pro, 2003 Server. Uneffected Platforms: Unix, MacOS (?) History: During the week of 2/9/04 the call center started recieving reports of users being unable to connect to sites off the CityNet network. Sites hosted on the internal network are uneffected by the outage. Initally it was thought to be a Internet Explorer problem possably caused by the KB832894 / IE SP1 or other updates but after further investigation it was found that Mozilla users were encountering the same problem. After several days of testing it was determined that during the outage any TCP session started on any port would fail. TCP sessions started before the outage continue to work and show no ill effects from the outage. After logging connection attempts at various intervals on many machines there appears to be no sort of pattern in the outages. Most machines encounter the problem, some more than others and a few do not encounter it at all. The duration and frequency of the outage is very fluid. During an outage, we can verify that the packet does seem to leave and reenter the network: Mar 5 22:28:04 pittpa-chaswv-ds3 17083: SLOT 2:6d20h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.14.174(3376) - 216.41.224.3(80), 1 packet Mar 5 22:28:09 pittpa-chaswv-ds3 17084: SLOT 1:6d20h: %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) - 69.43.14.174(3376), 1 packet Mar 5 22:28:09 pittpa-chaswv-ds3 17085: SLOT 2:6d20h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.14.174(3378) - 216.41.224.3(80), 1 packet Mar 5 22:28:09 pittpa-chaswv-ds3 17086: SLOT 1:6d20h: %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) - 69.43.14.174(3378), 1 packet Mar 5 22:33:24 pittpa-chaswv-ds3 17089: SLOT 1:6d20h: %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) - 69.43.14.174(3378), 7 packets Mar 5 22:33:24 pittpa-chaswv-ds3 17090: SLOT 1:6d20h: %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) - 69.43.14.174(3376), 17 packets Mar 5 22:33:58 pittpa-chaswv-ds3 17092: SLOT 2:6d20h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.14.174(3378) - 216.41.224.3(80), 7 packets Mar 5 22:33:58 pittpa-chaswv-ds3 17093: SLOT 2:6d20h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.14.174(3376) - 216.41.224.3(80), 18 packets Mar 5 00:58:30 pittpa-clarwv-ds3 16062: SLOT 2:5d22h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3183) - 216.41.224.3(80), 1 packet Mar 5 00:58:30 pittpa-clarwv-ds3 16063: SLOT 1:5d22h: %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) - 69.43.23.23(3183), 1 packet Mar 5 01:03:28 pittpa-clarwv-ds3 16067: SLOT 2:5d22h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3217) - 216.41.224.3(80), 1 packet Mar 5 01:03:28 pittpa-clarwv-ds3 16068: SLOT 1:5d22h: %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) - 69.43.23.23(3217), 1 packet Mar 5 01:03:34 pittpa-clarwv-ds3 16069: SLOT 2:5d22h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3228) - 216.41.224.3(80), 1 packet Mar 5 01:03:34 pittpa-clarwv-ds3 16070: SLOT 1:5d22h: %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) - 69.43.23.23(3228), 1 packet Mar 5 01:03:39 pittpa-clarwv-ds3 16072: SLOT 2:5d22h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3239) - 216.41.224.3(80), 1 packet Mar 5 01:03:47 pittpa-clarwv-ds3 16073: SLOT 2:5d22h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3183) - 216.41.224.3(80), 74 packets Mar 5 01:04:13 pittpa-clarwv-ds3 16075: SLOT 1:5d22h: %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80
Re: FW: hey had eric sent you
On Fri, 2004-03-12 at 21:17, Riley, Marty wrote: 10:17:16.416222 IP 192.168.1.1.1900 239.255.255.250.1900: udp 278 This is UPnP discovery. Take a look here: http://www.nthelp.com/upnpscrewup.htm http://www.linuxsa.org.au/mailing-list/2002-11/1134.html I see a lot of unicast UPnP traffic on my networks. UPnP seems like a train wreck waiting to happen, to me. It would be interesting to see what happens if one of your users turns UPnP off on their host. Just a shot in the dark. -- James H. Edwards Routing and Security At the Santa Fe Office: Internet at Cyber Mesa [EMAIL PROTECTED] [EMAIL PROTECTED]
Re: hey had eric sent you
MessageThis in reply to the earlier thread Weird Problems? Well barring that, I've seen simuliar issues, maybe not the exact same timings but. I've noticed a couple of things while working with a roll out of Active-Directory and a recent upgrade to I.E 6.0 for the user base. Since there were several thousand users involved some of the issues were simply bad configs/drivers/etc. However one of the stats I have noticed is that in certain situations where a system is connected to a Cisco 3548, and the client is running in an Auto detect (AD/AN) mode that things are horendiously slow during boot up, and at various times seem to hang unexplainably. It seemed corrected by setting the client to 100/Full, but not in all cases. Lots of HTTP complaints still remain about accessing webpages etc. but never consistant. This of course is a pretty fresh problem and is still in my queue for research to start this Monday. As well, we've found that there was an odd bug with Cisco's 6513s and their 48 port 10/100/1000 line cards. This was using the latest IOS/CAT software at the time. Again not sure if its a documented problem but, several users were unable to Telnet or FTP to systems that teminated to the 6513, oddly we were able to icmp echo and pass HTTP. After sometime and 2 TACs I found that there was a bug regarding these items and real small packets (I Think less than 64bits??) being passed thru the 6513 and got an RMA for new Line cards. Again, perhaps nothing to do with your situation. Since the Nix systems and non-Doze seem not to have an issue, perhaps you can enlighten me with further Sniffs/Captures of these events directly? As soon as I get some more data/Captures on my end from the problems I'm seeing I can forward those apon request so as to keep S/N ratio down on Nanog (: Cheers, -Joe - Original Message - From: Riley, Marty To: [EMAIL PROTECTED] Sent: Friday, March 12, 2004 11:17 PM Subject: FW: hey had eric sent you
