Re: [tcpdump-workers] Libpcap performance problem
Hi, We are using version 1.6.2. Regards,Giray From: anders.bro...@ericsson.com To: tcpdump-workers@lists.tcpdump.org Date: Wed, 28 Jan 2015 15:21:10 + Subject: Re: [tcpdump-workers] Libpcap performance problem Hi, What version of libpcap are you using? Version = 1.5.3 uses TPACKET_V3 which may give an capacity increase. Regards Anders -Original Message- From: tcpdump-workers [mailto:tcpdump-workers-boun...@lists.tcpdump.org] On Behalf Of Giray Simsek Sent: den 28 januari 2015 15:58 To: tcpdump-workers@lists.tcpdump.org Subject: [tcpdump-workers] Libpcap performance problem Hi, We are currently working on testing Linux network performance. We have two Linux machines in our test setup. Machine1 is the attacker machine from which we are sending SYN packets to Machine2 at a rate of 3million pps. We are able to receive these packets on Machine2's external interface and forward them through the internal interface without dropping any packets. So far no problems. However, when we start another app that captures traffic on Machine2's external interface using libpcap, the amount of traffic that is forwarded drops significantly. Obviously, this second libpcap app becomes a bottleneck. It can capture only about 800Kpps of traffic and only about 800Kpps can be forwarded in this case. This drop in the amount of forwarded traffic is not acceptable for us. Is there any way we can overcome this problem? Are there any settings on Os, ixgbe driver or libpcap that will allow us to forward all the traffic? Both machines are running Linux kernel 3.15. Thanks in advance. Giray ___ tcpdump-workers mailing list tcpdump-workers@lists.tcpdump.org https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers ___ tcpdump-workers mailing list tcpdump-workers@lists.tcpdump.org https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers ___ tcpdump-workers mailing list tcpdump-workers@lists.tcpdump.org https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers
Re: [tcpdump-workers] Libpcap performance problem
We don't see any signs that indicate problems with skb allocations on 'top' output or in /var/log/messages.Also, if we set a filter that causes libpcap to return no packets, then we are not experiencing the problem.All of the traffic is forwarded with no drops. It's ok if we're not able to capture all of the 3Mpps using libpcap, in fact we can only capture around 1Mpps of the whole traffic.But what we don't understand is why it affects the rate of traffic that is forwarded through the internal interface significantly (by reducing it from 3Mpps to ~Mpps). And also we are wondering if there is a way to change this behaviour. That is read as much as you can with libpcap but don't block the forwarded traffic.We tried PF_RING as well and got similar results. Of course, we don't want to use the PF_RING in DNA mode as this will bypass the kernel. Regards,Giray From: david.lai...@aculab.com To: rick.jon...@hp.com; tcpdump-workers@lists.tcpdump.org Date: Wed, 28 Jan 2015 17:17:15 + Subject: Re: [tcpdump-workers] Libpcap performance problem From: Rick Jones On 01/28/2015 06:57 AM, Giray Simsek wrote: Hi, We are currently working on testing Linux network performance. We have two Linux machines in our test setup. Machine1 is the attacker machine from which we are sending SYN packets to Machine2 at a rate of 3million pps. We are able to receive these packets on Machine2's external interface and forward them through the internal interface without dropping any packets. So far no problems. However, when we start another app that captures traffic on Machine2's external interface using libpcap, the amount of traffic that is forwarded drops significantly. Obviously, this second libpcap app becomes a bottleneck. It can capture only about 800Kpps of traffic and only about 800Kpps can be forwarded in this case. This drop in the amount of forwarded traffic is not acceptable for us. Is there any way we can overcome this problem? Are there any settings on Os, ixgbe driver or libpcap that will allow us to forward all the traffic? Both machines are running Linux kernel 3.15. TCP SYN segments would be something like 66 bytes per (I'm assuming some options being set in the SYN). At 3 million packets per second, that would be 198 million bytes per second. Perhaps overly paranoid of me but can the storage on Machine2 keep-up with that without say the bulk of the RAM being taken-over by buffer cache and perhaps inhibiting skb alloctions? More likely is that running pcap requires that every receive packet be copied (so it can be delivered to pcap and IP). The cost of doing this could easily be significant. Even setting a pcap filter to return no packets will invoke the same overhead. As does running the dhcp client! David ___ tcpdump-workers mailing list tcpdump-workers@lists.tcpdump.org https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers ___ tcpdump-workers mailing list tcpdump-workers@lists.tcpdump.org https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers
[tcpdump-workers] Libpcap performance problem
Hi, We are currently working on testing Linux network performance. We have two Linux machines in our test setup. Machine1 is the attacker machine from which we are sending SYN packets to Machine2 at a rate of 3million pps. We are able to receive these packets on Machine2's external interface and forward them through the internal interface without dropping any packets. So far no problems. However, when we start another app that captures traffic on Machine2's external interface using libpcap, the amount of traffic that is forwarded drops significantly. Obviously, this second libpcap app becomes a bottleneck. It can capture only about 800Kpps of traffic and only about 800Kpps can be forwarded in this case. This drop in the amount of forwarded traffic is not acceptable for us. Is there any way we can overcome this problem? Are there any settings on Os, ixgbe driver or libpcap that will allow us to forward all the traffic? Both machines are running Linux kernel 3.15. Thanks in advance. Giray ___ tcpdump-workers mailing list tcpdump-workers@lists.tcpdump.org https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers
Re: [tcpdump-workers] Libpcap performance problem
On 01/28/2015 06:57 AM, Giray Simsek wrote: Hi, We are currently working on testing Linux network performance. We have two Linux machines in our test setup. Machine1 is the attacker machine from which we are sending SYN packets to Machine2 at a rate of 3million pps. We are able to receive these packets on Machine2's external interface and forward them through the internal interface without dropping any packets. So far no problems. However, when we start another app that captures traffic on Machine2's external interface using libpcap, the amount of traffic that is forwarded drops significantly. Obviously, this second libpcap app becomes a bottleneck. It can capture only about 800Kpps of traffic and only about 800Kpps can be forwarded in this case. This drop in the amount of forwarded traffic is not acceptable for us. Is there any way we can overcome this problem? Are there any settings on Os, ixgbe driver or libpcap that will allow us to forward all the traffic? Both machines are running Linux kernel 3.15. TCP SYN segments would be something like 66 bytes per (I'm assuming some options being set in the SYN). At 3 million packets per second, that would be 198 million bytes per second. Perhaps overly paranoid of me but can the storage on Machine2 keep-up with that without say the bulk of the RAM being taken-over by buffer cache and perhaps inhibiting skb alloctions? If you aren't trying to forward the SYNs and just let them bit-bucket, is the packet capture able to keep-up? rick jones ___ tcpdump-workers mailing list tcpdump-workers@lists.tcpdump.org https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers
Re: [tcpdump-workers] Libpcap performance problem
From: Rick Jones On 01/28/2015 06:57 AM, Giray Simsek wrote: Hi, We are currently working on testing Linux network performance. We have two Linux machines in our test setup. Machine1 is the attacker machine from which we are sending SYN packets to Machine2 at a rate of 3million pps. We are able to receive these packets on Machine2's external interface and forward them through the internal interface without dropping any packets. So far no problems. However, when we start another app that captures traffic on Machine2's external interface using libpcap, the amount of traffic that is forwarded drops significantly. Obviously, this second libpcap app becomes a bottleneck. It can capture only about 800Kpps of traffic and only about 800Kpps can be forwarded in this case. This drop in the amount of forwarded traffic is not acceptable for us. Is there any way we can overcome this problem? Are there any settings on Os, ixgbe driver or libpcap that will allow us to forward all the traffic? Both machines are running Linux kernel 3.15. TCP SYN segments would be something like 66 bytes per (I'm assuming some options being set in the SYN). At 3 million packets per second, that would be 198 million bytes per second. Perhaps overly paranoid of me but can the storage on Machine2 keep-up with that without say the bulk of the RAM being taken-over by buffer cache and perhaps inhibiting skb alloctions? More likely is that running pcap requires that every receive packet be copied (so it can be delivered to pcap and IP). The cost of doing this could easily be significant. Even setting a pcap filter to return no packets will invoke the same overhead. As does running the dhcp client! David ___ tcpdump-workers mailing list tcpdump-workers@lists.tcpdump.org https://lists.sandelman.ca/mailman/listinfo/tcpdump-workers
