Hi Alfredo, I can confirm the same results as Jesse. Pulling the latest svn (from Friday) produced no noticeable difference in drop behavior.
Thanks, Derek From: [email protected] [mailto:[email protected]] On Behalf Of Jesse Bowling Sent: Sunday, April 27, 2014 9:28 AM To: [email protected] Subject: Re: [Ntop-misc] Substantially higher packet loss moving from PF_RING 5.6.1 to 5.6.2 Hi Alfredo, Sorry for the delay on response. I have tried the SVN code, and I'm afraid that it did not make a discernible difference on my test box. Anything else I can do to help troubleshoot? Cheers, Jesse On Sun, Apr 6, 2014 at 7:21 PM, Alfredo Cardigliano <[email protected]<mailto:[email protected]>> wrote: Hi Jesse I reworked a bit the fragment hash, could you run some test with latest code from svn? Please let us know. Alfredo On 05 Apr 2014, at 02:31, Jesse Bowling <[email protected]<mailto:[email protected]>> wrote: I was reminded that my filter filter would catch all but the last fragmented packet, so I ran a new capture with: time tcpdump -i 'p6p1;p6p2' -nn -c 1000 'ip[6] & 0x20 !=0 or ip[6:2] & 0x1fff != 0' > /dev/null (Thanks Judy for the filter!) It took just over 61 minutes to collect 1000 fragmented packets on a link that averaged 1.3 Gb/s during that hour. I wouldn't consider that heavily fragmented traffic (although that's obviously a small sample time). Could there be an issue with removing fragments from the cluster queue? Cheers, Jesse On Fri, Apr 4, 2014 at 1:34 PM, Jesse Bowling <[email protected]<mailto:[email protected]>> wrote: Hi Luca, Thank you for your response. I'll look into getting a traffic capture, but I'm not hopeful. I'd argue against the fragmentation issue for two reasons: *) The change in packet loss was consistent with the change in PF_RING versions, and previous versions did not exhibit this behavior *) Filtering current traffic with tcpdump -i 'p6p1;p6p2' -nn 'ip[6] & 0x20 != 0' does not seem to reveal any unusual amount of fragmented packets (a dozen or so over 3 minutes). If I'm unable to provide pcaps, what's next best? Any debug logging I can turn on, or attach to debugger, etc? Cheers, Jesse On Fri, Apr 4, 2014 at 11:03 AM, Luca Deri <[email protected]<mailto:[email protected]>> wrote: Jesse I am traveling with my team this week, sorry for not being responsive. Would you be able to provide us a traffic sample you have? It looks you have strongly frgmented traffic Regards Luca On 02 Apr 2014, at 11:42, Jesse Bowling <[email protected]<mailto:[email protected]>> wrote: I've not seen any comments or updates on this issue. I'm still experiencing this issue and would appreciate any advice on how to troubleshoot, or at least some confirmation from other sites that they see this issue with 5.6.2 PF_RING? Thanks, Jesse On Tue, Mar 25, 2014 at 2:14 PM, Jesse Bowling <[email protected]<mailto:[email protected]>> wrote: Hello, Recently I updated my sensors to use PF_RING 5.6.2, after a good long while on an SVN version of 5.6.1. Initial results on a lightly loaded test box were good, however when the same setup was loaded onto more heavily loaded boxes we started seeing a substantial number of dropped packets. We use the ixgbe PF_RING aware drivers on RHEL 6. The /proc/net/pf_ring/info reports: PF_RING Version : 5.6.2 ($Revision: exported$) Total rings : 28 Standard (non DNA) Options Ring slots : 16384 Slot version : 15 Capture TX : No [RX only] IP Defragment : No Socket Mode : Standard Transparent mode : No [mode 2] Total plugins : 0 Cluster Fragment Queue : 32720 Cluster Fragment Discard : 69173446 The Cluster Fragment Discard numbers are much larger than usual. Additionally the NIC started reporting drops (less than 1%, but compared to the previous setup which dropped 0 packets at the NIC). Finally, the snort and argus processes running on top of this setup now report drops at a much higher rate than before. Previously snort ran with 1% or less drops, argus typically dropped only a few thousandths of a percent. Currently, snort is reporting drops of 15-20%, and argus is seeing roughly 15% drops as well. This is on a 10 Gb link with peaks of 1.8 Gb/sec with daytime averages of 1.4 Gb/sec. Similar, although smaller, increases in dropped packets was seen on another link that is much less loaded (300-500 Mb/sec, loss typically in the 5-10% range). Overall this is a substantial loss of performance. While I also upgraded snort and argus to newer versions at the same time as PF_RING, I find it less likely that both programs made changes to adversely affect packet capture rates at the same time. Are there any other reports of this happening? Anything I can do to help troubleshoot where this packet loss might be happening? I will file this on the bugzilla as well for tracking. Cheers, Jesse -- Jesse Bowling -- Jesse Bowling _______________________________________________ Ntop-misc mailing list [email protected]<mailto:[email protected]> http://listgateway.unipi.it/mailman/listinfo/ntop-misc _______________________________________________ Ntop-misc mailing list [email protected]<mailto:[email protected]> http://listgateway.unipi.it/mailman/listinfo/ntop-misc -- Jesse Bowling -- Jesse Bowling _______________________________________________ Ntop-misc mailing list [email protected]<mailto:[email protected]> http://listgateway.unipi.it/mailman/listinfo/ntop-misc _______________________________________________ Ntop-misc mailing list [email protected]<mailto:[email protected]> http://listgateway.unipi.it/mailman/listinfo/ntop-misc -- Jesse Bowling
_______________________________________________ Ntop-misc mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop-misc
