I truncated the tcpdump output, perhaps I should've noted that.
Here's the packet debug from the cisco router - plain as day it's not being
nice - at least to me ;-)
IP: tableid=0, s=1.1.1.1 (local), d=2.2.2.2 (Tunnel0), routed via FIB
IP: s=1.1.1.1 (local), d=2.2.2.2 (Tunnel0), g=172.29.1.1, len 1492, forward
UDP src=50769, dst=2060
IP: s=1.1.1.1 (local), d=2.2.2.2 (Tunnel0), len 1396, sending fragment
IP Fragment, Ident = 14766, fragment offset = 0
UDP src=50769, dst=2060
IP: s=1.1.1.1 (local), d=2.2.2.2 (Tunnel0), len 116, sending last fragment
IP Fragment, Ident = 14766, fragment offset = 1376
IP: tableid=0, s=1.1.1.1 (local), d=2.2.2.2 (Tunnel0), routed via FIB
IP: s=1.1.1.1 (local), d=2.2.2.2 (Tunnel0), g=172.29.1.1, len 1492, forward
UDP src=50769, dst=2060
IP: s=1.1.1.1 (local), d=2.2.2.2 (Tunnel0), len 1396, sending fragment
IP Fragment, Ident = 14766, fragment offset = 0
UDP src=50769, dst=2060
IP: s=1.1.1.1 (local), d=2.2.2.2 (Tunnel0), len 116, sending last fragment
IP Fragment, Ident = 14766, fragment offset = 1376
Two questions:
1.) WHY is Cisco not using the "appropriate" packet size based on MTU
2.) WHY is the host reporting the checksum bad - OR -
2a.) Why is nTop not reporting stats about this exporter if chksum is
OK?
G
-----Original Message-----
From: [email protected]
[mailto:[email protected]] On Behalf Of Rick Jones
Sent: Tuesday, May 31, 2011 4:33 PM
To: [email protected]
Subject: Re: [Ntop] Cisco netflow packet size - adjusting?
On Tue, 2011-05-31 at 16:05 -0500, Gary Gatten wrote:
> Some interesting notes. Assuming I agree for now, I adjusted my filter:
> tcpdump -vvv -nn -s 1524 host 1.2.3.4.
>
> Same results:
> tcpdump: listening on rl0, link-type EN10MB (Ethernet), capture size 1524
> bytes
> 16:34:41.421480 IP (tos 0x0, ttl 253, id 14766, offset 0, flags [+], proto:
> UDP (17), length: 1396) 1.2.3.4.50769 > 5.5.5.5.2060: [bad udp cksum e710!]
> UDP, length 1464
If that is all that is being captured, then something is swallowing the
second IP datagram fragment of the IP datagram carrying the UDP datagram
carrying the netflow PDU that describes the flows that Jack built...
> This capture was on a FBSD box (vs RHEL) and outside the PIX FW.
>
> The whole issue that brought me here is several exporters NOT showing
> up in ntop. The common thread was those with high enough flows that
> they "need" large packets. My guess is ntop is not seeing 70% of my
> netflow packets from remote offices due to this issue. Not sure how
> to tell, except maybe turn on debug for netflow and listen to only a
> single office, else it would be WAY too much data...
>
> Yes - if the MTU of the interface is n, it should not format a packet
> larger than n. Even M$ does this correctly!
You still haven't got data showing that anyone was actually trying to
send *packets* on the wire larger than the MTU. (aka, IIRC, giants) What
data there is, suggests that at least as far as UDP and IP over Ethernet
are concerned there are no *packets* (eg frames) being sent which are
larger than the MTU.
I believe we are having a Can vs May vs Should discussion, intertwined
with the responsibility and terminology of the various layers and
perhaps what constitutes a "packet."
Applications are allowed to ask UDP to send up to just shy of 64 KB of
data per message, regardless of MTU (May). They have the socket calls
etc with which to do so (Can). Now, that said, indeed it is not a Good
Idea (tm) for an application to send a message via UDP that would cause
IP to have to fragment the UDP datagram carrying the user's message
(Should not). But that is not an all-or-nothing "should not" - there is
leeway for considering circumstances. The main reason for the "should
not" is the "all or nothing" nature of IP fragmentation and reassembly -
IP does not retransmit, and all the fragments of an IP datagram must
arrive at the receiver to reassemble the datagram. An underlying packet
loss rate then is "amplified" into a larger IP datagram loss rate.
But, it is still *perfectly* *legal* (can and may) as far as any of the
specifications are concerned for an application to send messages larger
than link-local MTU at one time via UDP. UDP checks the user's message
against its maximum message size limit (16 bits including headers), and
adds the UDP header and passes to IP if that check passes. IP then
checks the resulting datagram against the link-local MTU, sees the
datagram is larger, and then fragments into N fragments each of which
fits in the MTU. Those are then passed to Ethernet, which sends the
frames on the wire - messages to datagrams, datagrams to frames, the
"packets" (frames) are all <= MTU in size.
Now, that does not mean that applications cannot use stack-specific
means to determine link-local MTU and adjust their behaviour, but such
applications must then be able to, in essence, do the same segmentation
and reassembly that TCP does. And, short of additional headers in the
application's PDU (eg netflow or sflow) to say "this PDU is incomplete,
wait for the rest of it" a protocol such as netflow or sflow (or DNS)
could have a damn difficult time of it if it were *required* to honor
the link-local MTU when using UDP.
rick
(OK, perhaps sflow/netflow isn't the *best* example, because their PDU's
often hold several, smaller, otherwise independent things, which could
be sent separately, but I hope the gist of what I'm trying to convey is
there)
>
> G
>
>
> -----Original Message-----
> From: [email protected]
> [mailto:[email protected]] On Behalf Of Rick Jones
> Sent: Tuesday, May 31, 2011 3:57 PM
> To: [email protected]
> Subject: Re: [Ntop] Cisco netflow packet size - adjusting?
>
> On Tue, 2011-05-31 at 15:20 -0500, Gary Gatten wrote:
> > Tcpdump extract: notice all the 1464 byte packets are F'd, others OK:
> >
> > 15:16:24.818262 IP (tos 0x0, ttl 252, id 14766, offset 0, flags [none],
> > proto: UDP (17), length: 1012) 10.15.26.200.50769 >
> > monpapp1.waddell.com.netflow-Regn15: [udp sum ok] UDP, length 984
> >
> > 15:16:28.822210 IP (tos 0x0, ttl 251, id 14766, offset 0, flags [+], proto:
> > UDP (17), length: 1396) 10.15.26.200.50769 >
> > monpapp1.waddell.com.netflow-Regn15: [bad udp cksum 55bc!] UDP, length 1464
> >
> > 15:16:39.822211 IP (tos 0x0, ttl 252, id 14766, offset 0, flags [none],
> > proto: UDP (17), length: 1060) 10.15.26.200.50769 >
> > monpapp1.waddell.com.netflow-Regn15: [udp sum ok] UDP, length 1032
> >
> > 15:16:46.822499 IP (tos 0x0, ttl 251, id 14766, offset 0, flags [+], proto:
> > UDP (17), length: 1396) 10.15.26.200.50769 >
> > monpapp1.waddell.com.netflow-Regn15: [bad udp cksum 6409!] UDP, length 1464
> >
> > 15:16:50.821026 IP (tos 0x0, ttl 251, id 14766, offset 0, flags [+], proto:
> > UDP (17), length: 1396) 10.15.26.200.50769 >
> > monpapp1.waddell.com.netflow-Regn15: [bad udp cksum 9a8e!] UDP, length 1464
> >
> > 15:16:55.819812 IP (tos 0x0, ttl 251, id 14766, offset 0, flags [+], proto:
> > UDP (17), length: 1396) 10.15.26.200.50769 >
> > monpapp1.waddell.com.netflow-Regn15: [bad udp cksum 2191!] UDP, length 1464
>
> I don't believe that tcpdump will do IP datagram fragment reassembly, so
> it cannot really validate the UDP checksum when the UDP datagram is
> being carried in a fragmented IP datagram. Of course, one might argue
> that when tcpdump has only the first fragment (the "+" means the More
> Fragments bit is set) it should not report anything about the UDP
> checksum in the first place. That is something you might bring-up to
> the tcpdump-workers mailing list. And btw, a filter expression matching
> on UDP port number (eg "port netflow" or "port sflow" will not capture
> the other fragments of the IP datagram carrying the UDP datagram
> because, as it is IP doing the fragmentation, the UDP header, which is
> just payload to IP, is not replicated.
>
> So, from that tcpdump output at least, you don't really know that the
> 1464 byte packets were indeed "F'd" as you so eloquently put it :) You
> need to see UDP checksum failures reported by UDP at the receiver (eg in
> something like netstat statistics) after IP fragment reassembly has
> completed and the entire datagram has been passed to UDP. And/or IP
> fragment reassembly errors increasing.
>
> Also, from your previous message:
>
> > By "not honor": If the traffic is generated by the local system, it
> > should look at the MTU of the output interface before forming a packet
> > and "honor" the MTU size of said interface. I have set the source and
> > dest interface to have an MTU of 1400, yet it still sends 1464 byte
> > packets. Not very nice!
>
> I assume you mean - sends it unfragmented right? It is perfectly
> "legal" (if not necessarily advisable) for an application using UDP to
> send messages larger than the link-local MTU.
>
> FWIW, that 1464 in the tcpdump output above is the *UDP* datagram
> length. The length of the IP datagram is earlier in the line, and
> coupled with the "[+]" it reporting 1396 does suggest that the IP
> datagram carrying the UDP datagram was fragmented
>
> rick
>
> > -----Original Message-----
> > From: [email protected]
> > [mailto:[email protected]] On Behalf Of Rick Jones
> > Sent: Tuesday, May 31, 2011 2:48 PM
> > To: [email protected]
> > Subject: Re: [Ntop] Cisco netflow packet size - adjusting?
> >
> > On Tue, 2011-05-31 at 14:37 -0500, Gary Gatten wrote:
> > > Hello,
> > >
> > > Anyone know how to force the Cisco IOS, specifically netflow, to use
> > > packet sizes smaller than 1464 bytes? It doesn't seem to be honoring
> > > the MTU size of the interfaces. It gets fragmented and then Checksum
> > > is invalid, and then ntop doesn't see the packets. Annoying....
> >
> > Um, gets fragmented by *what* and where? Fragmentation isn't *supposed*
> > to corrupt datagrams, by what is the checksum being invalid being
> > reported?
> >
> > Also, whence 1464? The maximum UDP payload on a standard 1500 byte MTU
> > Ethernet-like network is 1472 bytes before the IPv4 datagram carrying
> > the UDP datagram carrying the user message has to be fragmented. Does
> > your network have a smaller than 1500 byte MTU?
> >
> > While IP fragmentation is indeed frowned upon, "not honoring the MTU
> > size" is a bit strong - UDP-using applications can indeed send messages
> > large enough to require fragmentation by IP. For example EDNS in DNS
> > will do so, and it isn't considered dishonorable :)
> >
> > rick jones
> >
> > > I've googled for hours and haven't found anything yet.
> > >
> > > TIA
> > >
> > > Gary
> > >
> > > Tags: netflow, ntop, exporter not recognized, cant see netflow traffic
> > >
> > >
> > >
> > > "This email is intended to be reviewed by only the intended recipient
> > > and may contain information that is privileged and/or confidential. If
> > > you are not the intended recipient, you are hereby notified that any
> > > review, use, dissemination, disclosure or copying of this email and
> > > its attachments, if any, is strictly prohibited. If you have received
> > > this email in error, please immediately notify the sender by return
> > > email and delete this email from your system."
> > > _______________________________________________
> > > Ntop mailing list
> > > [email protected]
> > > http://listgateway.unipi.it/mailman/listinfo/ntop
> >
> >
> > _______________________________________________
> > Ntop mailing list
> > [email protected]
> > http://listgateway.unipi.it/mailman/listinfo/ntop
> >
> >
> >
> >
> >
> > <font size="1">
> > <div style='border:none;border-bottom:double windowtext 2.25pt;padding:0in
> > 0in 1.0pt 0in'>
> > </div>
> > "This email is intended to be reviewed by only the intended recipient
> > and may contain information that is privileged and/or confidential.
> > If you are not the intended recipient, you are hereby notified that
> > any review, use, dissemination, disclosure or copying of this email
> > and its attachments, if any, is strictly prohibited. If you have
> > received this email in error, please immediately notify the sender by
> > return email and delete this email from your system."
> > </font>
> >
> > _______________________________________________
> > Ntop mailing list
> > [email protected]
> > http://listgateway.unipi.it/mailman/listinfo/ntop
>
>
> _______________________________________________
> Ntop mailing list
> [email protected]
> http://listgateway.unipi.it/mailman/listinfo/ntop
>
>
>
>
>
> <font size="1">
> <div style='border:none;border-bottom:double windowtext 2.25pt;padding:0in
> 0in 1.0pt 0in'>
> </div>
> "This email is intended to be reviewed by only the intended recipient
> and may contain information that is privileged and/or confidential.
> If you are not the intended recipient, you are hereby notified that
> any review, use, dissemination, disclosure or copying of this email
> and its attachments, if any, is strictly prohibited. If you have
> received this email in error, please immediately notify the sender by
> return email and delete this email from your system."
> </font>
>
> _______________________________________________
> Ntop mailing list
> [email protected]
> http://listgateway.unipi.it/mailman/listinfo/ntop
_______________________________________________
Ntop mailing list
[email protected]
http://listgateway.unipi.it/mailman/listinfo/ntop
<font size="1">
<div style='border:none;border-bottom:double windowtext 2.25pt;padding:0in 0in
1.0pt 0in'>
</div>
"This email is intended to be reviewed by only the intended recipient
and may contain information that is privileged and/or confidential.
If you are not the intended recipient, you are hereby notified that
any review, use, dissemination, disclosure or copying of this email
and its attachments, if any, is strictly prohibited. If you have
received this email in error, please immediately notify the sender by
return email and delete this email from your system."
</font>
_______________________________________________
Ntop mailing list
[email protected]
http://listgateway.unipi.it/mailman/listinfo/ntop
