Re: [tcpdump-workers] mmap consumes more CPU
> hi, > I just checked the two mechanism : > (1) Using mmap to fetch packets from kernel to userspace > (2) Using recvfrom() call to fetch packets > > I see top reports > (1) 34% memory 20% cpu usage > (2) 21% memory 7% cpu usage ! It is worth remembering that the cpu usage reported by top isn't worth the paper it is printed on for many workloads. IIRC it is based on the cpu state when the timer interrupt fires. processes that are scheduled very often, and run for short periods tend to get mis-counted. Since the Linux scheduler doesn't get a high-res timestamp everytime it does a process switch, about the only way to measure idle time is to put a very low priority process into a counting loop. Unfortunately the scheduler might make it difficult to make the processes priority low enough. David ___ tcpdump-workers mailing list tcpdump-workers@lists.tcpdump.org https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers
Re: [tcpdump-workers] vlan tagged packets and libpcap breakage
Daniel Borkmann writes: > Speaking of netsniff-ng where we don't reconstruct VLAN headers, users > have reported that depending on the NIC/driver resp. ethtool setting, > they can come in stripped or not (in the pf_packet's rx_ring buffer). > However, I assume VLAN AUXDATA is always consistent (and so the > BPF/BPF JIT filtering). Yes it was a mess before we added software stripping of the vlan headers a year ago. Eric ___ tcpdump-workers mailing list tcpdump-workers@lists.tcpdump.org https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers
Re: [tcpdump-workers] vlan tagged packets and libpcap breakage
Michael Richardson writes:
> Thank you for this reply.
>
>> "Eric" == Eric W Biederman writes:
> Eric> I don't see any need to add any kernel code to allow checking
> Eric> if vlan tags are stripped. Vlan headers are stripped on all
> Eric> kernel interfaces today. Vlan headers have been stripped on
> Eric> all but a handful of software interfaces for 6+ years. For
> Eric> all kernels if the vlan header is stripped it is reported in
> Eric> the auxdata, upon packet reception. Careful code should also
> Eric> look for TP_STATUS_VLAN_VALID which allows for distinguishing
> Eric> a striped vlan header of 0 from no vlan header.
>
> I can regularly see vlan tags on raw dumps from the untagged ("eth0")
> today, running 3.2 (debian stable):
>
> obiwan-[~] mcr 4848 %sudo tcpdump -i eth0 -n -p -e | grep -i vlan
> listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
> 17:05:15.404909 38:60:77:38:e6:47 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q
> (0x8100), length 46: vlan 3800, p 0, ethertype ARP, Request who-has
> 172.30.42.1 tell 172.30.42.11, length 28
>
> So, I'm curious about the statement that vlan tags have been stripped
> for some time, because I don't see them stripped today. My desktop has
> an Intel 82579V NIC in it...
So I just took a quick look at libpcap 1.1.1.1.
In pcap_read_packet in pcap-linux.c the code to readd the vlan header is
protected by HAVE_PACKET_AUXDATA.
In pcap_read_linux_mmap in pcap-linux.c the code to readd the vlan
header is protected by HAVE_TPACKET2.
That code is shifting the the ethernet header up 4 bytes and inserting
the vlan header in packets as we receive them.
The code is correct except for the case of packets in vlan 0. Currently
the packet reconstruction is ambiguous. The most recent kernels have
a TP_STATUS_VLAN_VALID flag that can be checked to see if the packet was
in vlan 0 or if there was no vlan at all. libpcap probably should be
taught how to handle TP_STATUS_VLAN_VALID so that it can get the vlan 0
handling correct.
> Eric> For old kernels that do not support the new extensions it is
> Eric> possible to generate code that looks at the ethernet header
> Eric> and sees if the ethertype is 0x8100 and then does things with
> Eric> it, but that will only work on a small handful of software
> Eric> only interfaces.
>
> at tcpdump.org, our concern is to release code that works on both new,
> and what for kernel.org folks would be considered "ancient" systems,
> such as Centos5/RHEL5 machines which are regularly still in production
> in the field (sadly...), but often need the latest diagnostics.
> What I hear you saying is that our existing code will work on older
> kernels, and that once we have new code to use the VLAN tag extensions,
> we should simply attempt to load it, and either it loads, or we get an
> error, and we go back to the code we had before. That's great news.
The existing code will work as well as anything if you don't have the
VLAN tag extensions. If you are not using the VLAN tag extensions there
is no reliable way to filter for vlan tagged packets in the kernel as on
most network devices (all for the last year) the vlan header has been
stripped at the time the bpf filter is run.
So yes. Just falling back to the existing code seems as good as it is
going to get. Which isn't very good but it took 5+ years before people
cared enough to get this fixed. :(
> The major concern is that if the 802.1q header is gone, although we can
> retrieve it somehow (I'm not sure how the AUXDATA is presented on the
> MMAP PF_PACKET interface...) we will have to reconstruct it in order to
> save it properly to a savefile. Many entities, including most major
> Network Telescopes (http://en.wikipedia.org/wiki/Network_telescope) use
> libpcap (and often tcpdump itself) to capture packets on probe points,
> and having good performance matters here...
Yes. I wasn't looking at the mmap path. The vlan information is
present in the tpacket2_hdr and tpacket3_hdr structures that accompany
packets read with the mmap interface. The old tpacket1_hdr doesn't
support the vlan_tci information so that won't work.
Not having the vlan header on the packet when the bpf filter is run is
justified by performance, and likewise reading vlan tagged packets to
userspace with the vlan header stripped is justified by performance.
Eric
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers
Re: [tcpdump-workers] vlan tagged packets and libpcap breakage
On Sat, Nov 17, 2012 at 11:14 PM, Michael Richardson wrote:
>
> Thank you for this reply.
>
>> "Eric" == Eric W Biederman writes:
> Eric> I don't see any need to add any kernel code to allow checking
> Eric> if vlan tags are stripped. Vlan headers are stripped on all
> Eric> kernel interfaces today. Vlan headers have been stripped on
> Eric> all but a handful of software interfaces for 6+ years. For
> Eric> all kernels if the vlan header is stripped it is reported in
> Eric> the auxdata, upon packet reception. Careful code should also
> Eric> look for TP_STATUS_VLAN_VALID which allows for distinguishing
> Eric> a striped vlan header of 0 from no vlan header.
>
> I can regularly see vlan tags on raw dumps from the untagged ("eth0")
> today, running 3.2 (debian stable):
>
> obiwan-[~] mcr 4848 %sudo tcpdump -i eth0 -n -p -e | grep -i vlan
> listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
> 17:05:15.404909 38:60:77:38:e6:47 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q
> (0x8100), length 46: vlan 3800, p 0, ethertype ARP, Request who-has
> 172.30.42.1 tell 172.30.42.11, length 28
>
> So, I'm curious about the statement that vlan tags have been stripped
> for some time, because I don't see them stripped today. My desktop has
> an Intel 82579V NIC in it...
Speaking of netsniff-ng where we don't reconstruct VLAN headers, users
have reported that depending on the NIC/driver resp. ethtool setting,
they can come in stripped or not (in the pf_packet's rx_ring buffer).
However, I assume VLAN AUXDATA is always consistent (and so the
BPF/BPF JIT filtering).
> Eric> For old kernels that do not support the new extensions it is
> Eric> possible to generate code that looks at the ethernet header
> Eric> and sees if the ethertype is 0x8100 and then does things with
> Eric> it, but that will only work on a small handful of software
> Eric> only interfaces.
>
> at tcpdump.org, our concern is to release code that works on both new,
> and what for kernel.org folks would be considered "ancient" systems,
> such as Centos5/RHEL5 machines which are regularly still in production
> in the field (sadly...), but often need the latest diagnostics.
>
> What I hear you saying is that our existing code will work on older
> kernels, and that once we have new code to use the VLAN tag extensions,
> we should simply attempt to load it, and either it loads, or we get an
> error, and we go back to the code we had before. That's great news.
Yes, this should be handled in such a way.
___
tcpdump-workers mailing list
tcpdump-workers@lists.tcpdump.org
https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers
Re: [tcpdump-workers] vlan tagged packets and libpcap breakage
Ani Sinha writes: > cc'ing netdev. > > > On Wed, Oct 31, 2012 at 2:01 PM, Michael Richardson wrote: >> >> Thanks for this email. >> >>> "Ani" == Ani Sinha writes: >> Ani> remove "inline" from vlan_core.c functions >> Ani> Signed-off-by: David S. Miller >> >> Ani> As a result of this patch, with or without HW acceleration support >> in >> Ani> the kernel driver, the vlan tag information is pulled out from the >> Ani> packet and is inserted in the skb metadata. Thereafter, the vlan tag >> Ani> information for a 802.1q packet can be obtained my applications >> Ani> running in the user space by using the auxdata and cmsg >> Ani> structure. >> >> Do you think that the existance of this behaviour could be exposed in >> sysctl, /proc/net or /sys equivalent (we still don't have /sys/net...)? >> As a read only file that had a 0/1 in it? > > yes, we definitely need a run time check. Whether this could be in the > form of a socket option or a /proc entry I don't know. I don't see any need to add any kernel code to allow checking if vlan tags are stripped. Vlan headers are stripped on all kernel interfaces today. Vlan headers have been stripped on all but a handful of software interfaces for 6+ years. For all kernels if the vlan header is stripped it is reported in the auxdata, upon packet reception. Careful code should also look for TP_STATUS_VLAN_VALID which allows for distinguishing a striped vlan header of 0 from no vlan header. The safe assumption then is that testing for vlan headers and vlan values in bpf filters is not possible without the new bpf extentions. It is possible to test for the presence of support of the new vlan bpf extensions by attempting to load a filter that uses them. As only valid filters can be loaded, old kernels that do not support filtering of vlan tags will fail to load the a test filter with uses them. For old kernels that do not support the new extensions it is possible to generate code that looks at the ethernet header and sees if the ethertype is 0x8100 and then does things with it, but that will only work on a small handful of software only interfaces. Eric ___ tcpdump-workers mailing list tcpdump-workers@lists.tcpdump.org https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers
Re: [tcpdump-workers] [PATCH] Decode DHCPv6 AFTR-Name option (RFC6334)
Hello, Please find the patch version 2 as an attachment. Updated for more tests on lengths. Any comment is welcome. Francois-Xavier - Original Message - > From: François-Xavier Le Bail > To: "tcpdump-workers@lists.tcpdump.org" > Cc: "fx.leb...@yahoo.com" > Sent: Wednesday, November 14, 2012 6:19 PM > Subject: [PATCH] Decode DHCPv6 AFTR-Name option (RFC6334) > > Hello, > > Please find the patch as an attachment. > > Any comment is welcome. > > Francois-Xavier Le Bail > ___ tcpdump-workers mailing list tcpdump-workers@lists.tcpdump.org https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers
[tcpdump-workers] [PATCH] Decode DHCPv6 AFTR-Name option (RFC6334)
Hello, Please find the patch as an attachment. Any comment is welcome. Francois-Xavier Le Bail ___ tcpdump-workers mailing list tcpdump-workers@lists.tcpdump.org https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers
