On Tue, 2011-05-31 at 19:15 -0500, Gary Gatten wrote: > I could open a TAC case, MAYBE 3 years from now there will be a fix.
Switch vendors?-) > ... 80014 is funny. Actual HP part. I have an original and I make the occasional replica for departing/retiring HPers, using a stamp set, a gold-ink stamp-pad and some spray varnish. > SPAN traces all look good, the BS fragmented packets are at least > getting to the end switch port correctly. So, why does the host OS > hose them: it is a VMWare guest, so MAYBE it's hosing things. Or, we > can assume the packets gets their fine, the OS likes them, but ntop > doesn't like them! So - why does nTop hate me and my BS packets? Perhaps ntop shares my more nuanced view of IP fragmentation and can sense your hostility towards it?-) That said, of course ntop won't know from IP fragmentation - it will only receive entire netflow PDUs post-reassembly by IP and acceptance of checksum by UDP. Strace may be your friend here. Particularly if the vmware hypervisor and VM guest aren't running a firewall of some sort. rick > More discovery needed, tomorrow is another day. > > On another note, if nTop supported netflow listeners on the SCTP protocol - > that would *maybe* be a good thing. > > I can build an instance with all the ntop netflow debugs enabled and > run it listening on a "test" port to minimize the noise. As noted, > tomorrow is another day... > > G > > > -----Original Message----- > From: [email protected] > [mailto:[email protected]] On Behalf Of Rick Jones > Sent: Tuesday, May 31, 2011 6:10 PM > To: [email protected] > Subject: Re: [Ntop] Cisco netflow packet size - adjusting? > > On Tue, 2011-05-31 at 17:48 -0500, Gary Gatten wrote: > > This is the output from the source of the netflow packets; so not much > > goodness - just badness :) > > Clearly I need to visit and apply HP P/N 19511-80014 :) Another web > search for you. > > > The datagrams were perhaps 3 seconds apart. > > > > RCVQ's are fine except for when ntop FREAKS out - see other post. Right > > now the are zero. > > > > This is happening on 18xx and 28xx routers across several different > > versions of IOS. I even erased my netflow confs, set the MTU on the > > interface, and reapplied netflow - still sending packets greater than > > MTU size. > > At this point I suspect you need to fire-up the Cisco support contract > and ask them about configuring the maximum netflow PDU size. And while > you are there, ask them why they are using the same IP datagram ID like > that - admittedly there is a 1 in 65536 chance of that being a > coincidence but I doubt it. > > rick > > > I will use wireshark on a span'ed port and see if that shows any difference. > > > > G > > > > > > > > > > -----Original Message----- > > From: [email protected] > > [mailto:[email protected]] On Behalf Of Rick Jones > > Sent: Tuesday, May 31, 2011 5:38 PM > > To: [email protected] > > Subject: Re: [Ntop] Cisco netflow packet size - adjusting? > > > > On Tue, 2011-05-31 at 17:19 -0500, Gary Gatten wrote: > > > I truncated the tcpdump output, perhaps I should've noted that. > > > > It would have been helpful, yes. > > > > > Here's the packet debug from the cisco router - plain as day it's not > > > being nice - at least to me ;-) > > > > > > IP: tableid=0, s=1.1.1.1 (local), d=2.2.2.2 (Tunnel0), routed via FIB > > > IP: s=1.1.1.1 (local), d=2.2.2.2 (Tunnel0), g=172.29.1.1, len 1492, > > > forward > > > UDP src=50769, dst=2060 > > > IP: s=1.1.1.1 (local), d=2.2.2.2 (Tunnel0), len 1396, sending fragment > > > IP Fragment, Ident = 14766, fragment offset = 0 > > > UDP src=50769, dst=2060 > > > IP: s=1.1.1.1 (local), d=2.2.2.2 (Tunnel0), len 116, sending last fragment > > > IP Fragment, Ident = 14766, fragment offset = 1376 > > > > OK, both fragments of the IP datagram carrying the UDP datagram carrying > > the netflow PDU are getting as far as your router. Goodness so far. > > > > > > > IP: tableid=0, s=1.1.1.1 (local), d=2.2.2.2 (Tunnel0), routed via FIB > > > IP: s=1.1.1.1 (local), d=2.2.2.2 (Tunnel0), g=172.29.1.1, len 1492, > > > forward > > > UDP src=50769, dst=2060 > > > IP: s=1.1.1.1 (local), d=2.2.2.2 (Tunnel0), len 1396, sending fragment > > > IP Fragment, Ident = 14766, fragment offset = 0 > > > UDP src=50769, dst=2060 > > > IP: s=1.1.1.1 (local), d=2.2.2.2 (Tunnel0), len 116, sending last fragment > > > IP Fragment, Ident = 14766, fragment offset = 1376 > > > > Although there is a small bit of badness IMO in that 1.1.1.1 seems to be > > using the same IP datagram ID. Assuming that is not a cut-and-paste > > anomaly, how far apart in time are those two IP datagrams (or four > > fragments)? To make you even more of a "fan" of IP datagram > > fragmentation and reassembly, you might web search for "rick jones > > frankengrams" - the description is too long for this list :) > > > > > Two questions: > > > 1.) WHY is Cisco not using the "appropriate" packet size based on MTU > > > > Not that I seek to defend Cisco, but I think you and I will just have to > > agree to disagree on the definition of appropriate here :) > > > > > 2.) WHY is the host reporting the checksum bad - OR - > > > 2a.) Why is nTop not reporting stats about this exporter if chksum > > > is OK? > > > > If you are basing 2 on the tcpdump output, remember that reporting the > > checksum bad was just confusing behaviour on the part of tcpdump doing > > something with just the one fragment it should not have been doing. IE > > a bug of "misfeature" in tcpdump. So, ignore that. And perhaps, > > complain to your end system's stack provider that they should separate > > UDP checksum failures from UDP socket buffer overflows in the output of > > netstat :) > > > > As for 2a, do you, perhaps, have a firewall rule on the receiving > > end-system which blocks fragments? I believe that the taps used by > > libpcap come below the firewall in the stack, so tcpdump will show them > > arriving, but they won't get higher than the firewall software. > > > > Also, there is still the possibility that the ntop software is not > > keeping-up with draining the socket buffer - although in truth, unless > > it has set the socket buffer to be impossibly small, to the point of > > being smaller than the 1400-odd byte netflow PDUs, I would expect that > > to be affecting all sources more or less in proportion. Still, looking > > at "netstat -an | grep <portnumber>" and looking at the depth of the > > receive queue on the socket would be goodness. > > > > rick > > > > > > > > G > > > > > > > > > -----Original Message----- > > > From: [email protected] > > > [mailto:[email protected]] On Behalf Of Rick Jones > > > Sent: Tuesday, May 31, 2011 4:33 PM > > > To: [email protected] > > > Subject: Re: [Ntop] Cisco netflow packet size - adjusting? > > > > > > On Tue, 2011-05-31 at 16:05 -0500, Gary Gatten wrote: > > > > Some interesting notes. Assuming I agree for now, I adjusted my filter: > > > > tcpdump -vvv -nn -s 1524 host 1.2.3.4. > > > > > > > > Same results: > > > > tcpdump: listening on rl0, link-type EN10MB (Ethernet), capture size > > > > 1524 bytes > > > > 16:34:41.421480 IP (tos 0x0, ttl 253, id 14766, offset 0, flags [+], > > > > proto: UDP (17), length: 1396) 1.2.3.4.50769 > 5.5.5.5.2060: [bad udp > > > > cksum e710!] UDP, length 1464 > > > > > > If that is all that is being captured, then something is swallowing the > > > second IP datagram fragment of the IP datagram carrying the UDP datagram > > > carrying the netflow PDU that describes the flows that Jack built... > > > > > > > This capture was on a FBSD box (vs RHEL) and outside the PIX FW. > > > > > > > > The whole issue that brought me here is several exporters NOT showing > > > > up in ntop. The common thread was those with high enough flows that > > > > they "need" large packets. My guess is ntop is not seeing 70% of my > > > > netflow packets from remote offices due to this issue. Not sure how > > > > to tell, except maybe turn on debug for netflow and listen to only a > > > > single office, else it would be WAY too much data... > > > > > > > > Yes - if the MTU of the interface is n, it should not format a packet > > > > larger than n. Even M$ does this correctly! > > > > > > You still haven't got data showing that anyone was actually trying to > > > send *packets* on the wire larger than the MTU. (aka, IIRC, giants) What > > > data there is, suggests that at least as far as UDP and IP over Ethernet > > > are concerned there are no *packets* (eg frames) being sent which are > > > larger than the MTU. > > > > > > I believe we are having a Can vs May vs Should discussion, intertwined > > > with the responsibility and terminology of the various layers and > > > perhaps what constitutes a "packet." > > > > > > Applications are allowed to ask UDP to send up to just shy of 64 KB of > > > data per message, regardless of MTU (May). They have the socket calls > > > etc with which to do so (Can). Now, that said, indeed it is not a Good > > > Idea (tm) for an application to send a message via UDP that would cause > > > IP to have to fragment the UDP datagram carrying the user's message > > > (Should not). But that is not an all-or-nothing "should not" - there is > > > leeway for considering circumstances. The main reason for the "should > > > not" is the "all or nothing" nature of IP fragmentation and reassembly - > > > IP does not retransmit, and all the fragments of an IP datagram must > > > arrive at the receiver to reassemble the datagram. An underlying packet > > > loss rate then is "amplified" into a larger IP datagram loss rate. > > > > > > But, it is still *perfectly* *legal* (can and may) as far as any of the > > > specifications are concerned for an application to send messages larger > > > than link-local MTU at one time via UDP. UDP checks the user's message > > > against its maximum message size limit (16 bits including headers), and > > > adds the UDP header and passes to IP if that check passes. IP then > > > checks the resulting datagram against the link-local MTU, sees the > > > datagram is larger, and then fragments into N fragments each of which > > > fits in the MTU. Those are then passed to Ethernet, which sends the > > > frames on the wire - messages to datagrams, datagrams to frames, the > > > "packets" (frames) are all <= MTU in size. > > > > > > Now, that does not mean that applications cannot use stack-specific > > > means to determine link-local MTU and adjust their behaviour, but such > > > applications must then be able to, in essence, do the same segmentation > > > and reassembly that TCP does. And, short of additional headers in the > > > application's PDU (eg netflow or sflow) to say "this PDU is incomplete, > > > wait for the rest of it" a protocol such as netflow or sflow (or DNS) > > > could have a damn difficult time of it if it were *required* to honor > > > the link-local MTU when using UDP. > > > > > > rick > > > > > > (OK, perhaps sflow/netflow isn't the *best* example, because their PDU's > > > often hold several, smaller, otherwise independent things, which could > > > be sent separately, but I hope the gist of what I'm trying to convey is > > > there) > > > > > > > > > > > G > > > > > > > > > > > > -----Original Message----- > > > > From: [email protected] > > > > [mailto:[email protected]] On Behalf Of Rick Jones > > > > Sent: Tuesday, May 31, 2011 3:57 PM > > > > To: [email protected] > > > > Subject: Re: [Ntop] Cisco netflow packet size - adjusting? > > > > > > > > On Tue, 2011-05-31 at 15:20 -0500, Gary Gatten wrote: > > > > > Tcpdump extract: notice all the 1464 byte packets are F'd, others OK: > > > > > > > > > > 15:16:24.818262 IP (tos 0x0, ttl 252, id 14766, offset 0, flags > > > > > [none], proto: UDP (17), length: 1012) 10.15.26.200.50769 > > > > > > monpapp1.waddell.com.netflow-Regn15: [udp sum ok] UDP, length 984 > > > > > > > > > > 15:16:28.822210 IP (tos 0x0, ttl 251, id 14766, offset 0, flags [+], > > > > > proto: UDP (17), length: 1396) 10.15.26.200.50769 > > > > > > monpapp1.waddell.com.netflow-Regn15: [bad udp cksum 55bc!] UDP, > > > > > length 1464 > > > > > > > > > > 15:16:39.822211 IP (tos 0x0, ttl 252, id 14766, offset 0, flags > > > > > [none], proto: UDP (17), length: 1060) 10.15.26.200.50769 > > > > > > monpapp1.waddell.com.netflow-Regn15: [udp sum ok] UDP, length 1032 > > > > > > > > > > 15:16:46.822499 IP (tos 0x0, ttl 251, id 14766, offset 0, flags [+], > > > > > proto: UDP (17), length: 1396) 10.15.26.200.50769 > > > > > > monpapp1.waddell.com.netflow-Regn15: [bad udp cksum 6409!] UDP, > > > > > length 1464 > > > > > > > > > > 15:16:50.821026 IP (tos 0x0, ttl 251, id 14766, offset 0, flags [+], > > > > > proto: UDP (17), length: 1396) 10.15.26.200.50769 > > > > > > monpapp1.waddell.com.netflow-Regn15: [bad udp cksum 9a8e!] UDP, > > > > > length 1464 > > > > > > > > > > 15:16:55.819812 IP (tos 0x0, ttl 251, id 14766, offset 0, flags [+], > > > > > proto: UDP (17), length: 1396) 10.15.26.200.50769 > > > > > > monpapp1.waddell.com.netflow-Regn15: [bad udp cksum 2191!] UDP, > > > > > length 1464 > > > > > > > > I don't believe that tcpdump will do IP datagram fragment reassembly, so > > > > it cannot really validate the UDP checksum when the UDP datagram is > > > > being carried in a fragmented IP datagram. Of course, one might argue > > > > that when tcpdump has only the first fragment (the "+" means the More > > > > Fragments bit is set) it should not report anything about the UDP > > > > checksum in the first place. That is something you might bring-up to > > > > the tcpdump-workers mailing list. And btw, a filter expression matching > > > > on UDP port number (eg "port netflow" or "port sflow" will not capture > > > > the other fragments of the IP datagram carrying the UDP datagram > > > > because, as it is IP doing the fragmentation, the UDP header, which is > > > > just payload to IP, is not replicated. > > > > > > > > So, from that tcpdump output at least, you don't really know that the > > > > 1464 byte packets were indeed "F'd" as you so eloquently put it :) You > > > > need to see UDP checksum failures reported by UDP at the receiver (eg in > > > > something like netstat statistics) after IP fragment reassembly has > > > > completed and the entire datagram has been passed to UDP. And/or IP > > > > fragment reassembly errors increasing. > > > > > > > > Also, from your previous message: > > > > > > > > > By "not honor": If the traffic is generated by the local system, it > > > > > should look at the MTU of the output interface before forming a packet > > > > > and "honor" the MTU size of said interface. I have set the source and > > > > > dest interface to have an MTU of 1400, yet it still sends 1464 byte > > > > > packets. Not very nice! > > > > > > > > I assume you mean - sends it unfragmented right? It is perfectly > > > > "legal" (if not necessarily advisable) for an application using UDP to > > > > send messages larger than the link-local MTU. > > > > > > > > FWIW, that 1464 in the tcpdump output above is the *UDP* datagram > > > > length. The length of the IP datagram is earlier in the line, and > > > > coupled with the "[+]" it reporting 1396 does suggest that the IP > > > > datagram carrying the UDP datagram was fragmented > > > > > > > > rick > > > > > > > > > -----Original Message----- > > > > > From: [email protected] > > > > > [mailto:[email protected]] On Behalf Of Rick Jones > > > > > Sent: Tuesday, May 31, 2011 2:48 PM > > > > > To: [email protected] > > > > > Subject: Re: [Ntop] Cisco netflow packet size - adjusting? > > > > > > > > > > On Tue, 2011-05-31 at 14:37 -0500, Gary Gatten wrote: > > > > > > Hello, > > > > > > > > > > > > Anyone know how to force the Cisco IOS, specifically netflow, to use > > > > > > packet sizes smaller than 1464 bytes? It doesn't seem to be > > > > > > honoring > > > > > > the MTU size of the interfaces. It gets fragmented and then > > > > > > Checksum > > > > > > is invalid, and then ntop doesn't see the packets. Annoying.... > > > > > > > > > > Um, gets fragmented by *what* and where? Fragmentation isn't > > > > > *supposed* > > > > > to corrupt datagrams, by what is the checksum being invalid being > > > > > reported? > > > > > > > > > > Also, whence 1464? The maximum UDP payload on a standard 1500 byte MTU > > > > > Ethernet-like network is 1472 bytes before the IPv4 datagram carrying > > > > > the UDP datagram carrying the user message has to be fragmented. Does > > > > > your network have a smaller than 1500 byte MTU? > > > > > > > > > > While IP fragmentation is indeed frowned upon, "not honoring the MTU > > > > > size" is a bit strong - UDP-using applications can indeed send > > > > > messages > > > > > large enough to require fragmentation by IP. For example EDNS in DNS > > > > > will do so, and it isn't considered dishonorable :) > > > > > > > > > > rick jones > > > > > > > > > > > I've googled for hours and haven't found anything yet. > > > > > > > > > > > > TIA > > > > > > > > > > > > Gary > > > > > > > > > > > > Tags: netflow, ntop, exporter not recognized, cant see netflow > > > > > > traffic > > > > > > > > > > > > > > > > > > > > > > > > "This email is intended to be reviewed by only the intended > > > > > > recipient > > > > > > and may contain information that is privileged and/or confidential. > > > > > > If > > > > > > you are not the intended recipient, you are hereby notified that any > > > > > > review, use, dissemination, disclosure or copying of this email and > > > > > > its attachments, if any, is strictly prohibited. If you have > > > > > > received > > > > > > this email in error, please immediately notify the sender by return > > > > > > email and delete this email from your system." > > > > > > _______________________________________________ > > > > > > Ntop mailing list > > > > > > [email protected] > > > > > > http://listgateway.unipi.it/mailman/listinfo/ntop > > > > > > > > > > > > > > > _______________________________________________ > > > > > Ntop mailing list > > > > > [email protected] > > > > > http://listgateway.unipi.it/mailman/listinfo/ntop > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > <font size="1"> > > > > > <div style='border:none;border-bottom:double windowtext > > > > > 2.25pt;padding:0in 0in 1.0pt 0in'> > > > > > </div> > > > > > "This email is intended to be reviewed by only the intended recipient > > > > > and may contain information that is privileged and/or confidential. > > > > > If you are not the intended recipient, you are hereby notified that > > > > > any review, use, dissemination, disclosure or copying of this email > > > > > and its attachments, if any, is strictly prohibited. If you have > > > > > received this email in error, please immediately notify the sender by > > > > > return email and delete this email from your system." > > > > > </font> > > > > > > > > > > _______________________________________________ > > > > > Ntop mailing list > > > > > [email protected] > > > > > http://listgateway.unipi.it/mailman/listinfo/ntop > > > > > > > > > > > > _______________________________________________ > > > > Ntop mailing list > > > > [email protected] > > > > http://listgateway.unipi.it/mailman/listinfo/ntop > > > > > > > > > > > > > > > > > > > > > > > > <font size="1"> > > > > <div style='border:none;border-bottom:double windowtext > > > > 2.25pt;padding:0in 0in 1.0pt 0in'> > > > > </div> > > > > "This email is intended to be reviewed by only the intended recipient > > > > and may contain information that is privileged and/or confidential. > > > > If you are not the intended recipient, you are hereby notified that > > > > any review, use, dissemination, disclosure or copying of this email > > > > and its attachments, if any, is strictly prohibited. If you have > > > > received this email in error, please immediately notify the sender by > > > > return email and delete this email from your system." > > > > </font> > > > > > > > > _______________________________________________ > > > > Ntop mailing list > > > > [email protected] > > > > http://listgateway.unipi.it/mailman/listinfo/ntop > > > > > > > > > _______________________________________________ > > > Ntop mailing list > > > [email protected] > > > http://listgateway.unipi.it/mailman/listinfo/ntop > > > > > > > > > > > > > > > > > > <font size="1"> > > > <div style='border:none;border-bottom:double windowtext > > > 2.25pt;padding:0in 0in 1.0pt 0in'> > > > </div> > > > "This email is intended to be reviewed by only the intended recipient > > > and may contain information that is privileged and/or confidential. > > > If you are not the intended recipient, you are hereby notified that > > > any review, use, dissemination, disclosure or copying of this email > > > and its attachments, if any, is strictly prohibited. If you have > > > received this email in error, please immediately notify the sender by > > > return email and delete this email from your system." > > > </font> > > > > > > _______________________________________________ > > > Ntop mailing list > > > [email protected] > > > http://listgateway.unipi.it/mailman/listinfo/ntop > > > > > > _______________________________________________ > > Ntop mailing list > > [email protected] > > http://listgateway.unipi.it/mailman/listinfo/ntop > > > > > > > > > > > > <font size="1"> > > <div style='border:none;border-bottom:double windowtext 2.25pt;padding:0in > > 0in 1.0pt 0in'> > > </div> > > "This email is intended to be reviewed by only the intended recipient > > and may contain information that is privileged and/or confidential. > > If you are not the intended recipient, you are hereby notified that > > any review, use, dissemination, disclosure or copying of this email > > and its attachments, if any, is strictly prohibited. If you have > > received this email in error, please immediately notify the sender by > > return email and delete this email from your system." > > </font> > > > > _______________________________________________ > > Ntop mailing list > > [email protected] > > http://listgateway.unipi.it/mailman/listinfo/ntop > > > _______________________________________________ > Ntop mailing list > [email protected] > http://listgateway.unipi.it/mailman/listinfo/ntop > > > > > > <font size="1"> > <div style='border:none;border-bottom:double windowtext 2.25pt;padding:0in > 0in 1.0pt 0in'> > </div> > "This email is intended to be reviewed by only the intended recipient > and may contain information that is privileged and/or confidential. > If you are not the intended recipient, you are hereby notified that > any review, use, dissemination, disclosure or copying of this email > and its attachments, if any, is strictly prohibited. If you have > received this email in error, please immediately notify the sender by > return email and delete this email from your system." > </font> > > _______________________________________________ > Ntop mailing list > [email protected] > http://listgateway.unipi.it/mailman/listinfo/ntop _______________________________________________ Ntop mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop
