Hi Jim afaik you should use the “workers” mode (not sure what is your exact configuration now) in order to directly run one suricata processing thread for each zc queue, otherwise you still have suricata redistributing traffic from the queues (using zbalance for doing the distribution is useless in this case). Please also note that this works well as long as zbalance is able to properly load balance your traffic across the suricata queues (this depends on the IP distribution in your traffic). Please also try running pfcount on the ZC queues to make sure traffic is well balanced and check that zbalance is not the bottleneck.
Regards Alfredo > On 11 Aug 2016, at 21:27, Jim Hranicky <[email protected]> wrote: > > [ Including the thread from osif-users for context ] > > I'm testing out suricata and pfring. Currently, I'm running suri > using 22 pinned threads on the default 10G/82599ES interface (enp4s0), > and it's working fairly well. > > Suricata config : > > - interface: enp4s0 # Number of receive threads (>1 will enable > experimental flow pinned > # runmode) > threads: 22 > > I tried using 22 ZC queues, configuring suri like this: > > - interface: zc:99@0 > threads: 1 > - interface: zc:99@1 > threads: 1 > [...] > - interface: zc:99@21 > threads: 1 > > and zbalance like so: > > ./zbalance_ipc -i zc:enp4s0 -m 4 -n 22,1 -c 99 -g 0 -S 1 > > and it seemed to work, however, the performance was drastically reduced. > > With normal pf_ring (-i enp4s0) : > capture.kernel_packets | Total | > 13211759633 > capture.kernel_drops | Total | > 3867805 > > With 22 zc queues: > capture.kernel_packets | Total | > 92423279 > capture.kernel_drops | Total | > 619667008 > > I also tried just using one ZC queue, but found I seem to get better > performance > with the multi-threaded non-ZC version. > > Is there something I can do to get the benefits of both ZC and > multi-threading? > > Thanks, > > -- > Jim Hranicky > Data Security Specialist > UF Information Technology > 105 NW 16TH ST Room #104 GAINESVILLE FL 32603-1826 > 352-273-1341 > > -------- Forwarded Message -------- > Subject: Re: [Oisf-users] suricata with PF_RING Zero Copy/Pinned CPUs > Date: Thu, 11 Aug 2016 10:51:36 -0400 > From: Jim Hranicky <[email protected]> > To: Chris Wakelin <[email protected]>, > [email protected] > > Leaving off the -i seems work, thanks for your help. > > Unfortunately, performance drops significantly with this setup. > > With normal pf_ring (-i enp4s0) : > capture.kernel_packets | Total | > 13211759633 > capture.kernel_drops | Total | > 3867805 > > With 22 zc queues: > capture.kernel_packets | Total | > 92423279 > capture.kernel_drops | Total | > 619667008 > > I guess this is a question for the pfring list. > > Jim > > On 08/10/2016 09:08 PM, Jim Hranicky wrote: >> If you leave off the -i flag, will suri just use all the interfaces >> found in the config? >> >> Thanks for the response, >> Jim >> >> On 08/10/2016 07:40 PM, Chris Wakelin wrote: >>> I had this working when I still worked for the University of Reading >>> (which I left a year ago to join ET/Proofpoint). Alas I don't have >>> access to a machine running PF_RING at the moment. >>> >>> I did have a message thread in November 2014 about ZC + hugepages and >>> Suricata on the PF_RING (ntop-misc) mailing list >>> (http://lists.ntop.org/mailman/listinfo/ntop-misc - you need to >>> subscribe to see archives though). >>> >>> It looks like I had (uisng pfdnacluser_master rather than zbalance_ipc) >>> >>> insmod ixgbe.ko RSS=1,1 mtu=1522 adapters_to_enable=xx:xx:xx:xx:xx:xx >>> num_rx_slots=32768 num_tx_slots=0 numa_cpu_affinity=1,1 >>> ifconfig up dna0 >>> >>> echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages >>> cat /proc/meminfo | grep Huge >>> mount -t hugetlbfs none /mnt/huge >>> >>> pfdnacluster_master -i dna0 -c 1 -n 15,1 -r 15 -m 4 -u /mnt/huge -d >>> >>> I was running ARGUS (flow-capture) on dnacl:1@15 >>> >>> My Suricata config looked like (I know the cluster settings are ignored):- >>> >>> pfring: >>> - interface: dnacl:1@0 >>> threads: 1 >>> cluster-id: 99 >>> cluster-type: cluster_flow >>> - interface: dnacl:1@1 >>> threads: 1 >>> cluster-id: 99 >>> cluster-type: cluster_flow >>> >>> ... >>> >>> - interface: dnacl:1@14 >>> threads: 1 >>> cluster-id: 99 >>> cluster-type: cluster_flow >>> >>> >>> My problem with hugepages turned out to be having Suricata running as a >>> non-root user meant it didn't get the right permissions to access them. >>> I said I was going to investigate a fix in Suricata to drop privileges >>> later, but seems I never got around to it. I ran Suricata as root >>> instead as a workaround (obviously not ideal). I also had CPUs with 16 >>> real cores and hyperthreading disabled (that latter I now understand was >>> probably a bad idea!) >>> >>> Hope this gives some useful pointers, >>> Best Wishes, >>> Chris >>> >>> On 10/08/16 18:07, Jim Hranicky wrote: >>>> I'm able to run and get good results with using multiple threads >>>> on a pf-enabled interface when not running in ZC mode. I'm a little >>>> stumped though as to how to configure zbalance_ipc/suricata to use >>>> multiple threads using ZC. >>>> >>>> When run 1 queue for suri >>>> >>>> ./zbalance_ipc -i zc:enp4s0 -m 4 -n 1,1 -c 99 -g 0 -S 1 >>>> >>>> then specify the interface like so >>>> >>>> - interface: zc:99@0 >>>> threads: 22 >>>> >>>> and run this command >>>> >>>> /opt/suricata/bin/suricata -i zc:99@0 -c >>>> /opt/suricata/etc/suricata/suricata.yaml --pfring -vv >>>> >>>> I get this: >>>> >>>> 10/8/2016 -- 13:00:01 - <Perf> - (RX#01) Using PF_RING v.6.5.0, >>>> interface zc:99@0, cluster-id 1 >>>> >>>> 10/8/2016 -- 13:00:01 - <Error> - [ERRCODE: SC_ERR_PF_RING_OPEN(34)] - >>>> Failed to open zc:99@0: pfring_open error. Check if zc:99@0 exists and >>>> pf_ring module is loaded. >>>> >>>> 10/8/2016 -- 13:00:01 - <Error> - [ERRCODE: SC_ERR_PF_RING_OPEN(34)] - >>>> Failed to open zc:99@0: pfring_open error. Check if zc:99@0 exists and >>>> pf_ring module is loaded. >>>> >>>> 10/8/2016 -- 13:00:01 - <Error> - [ERRCODE: SC_ERR_PF_RING_OPEN(34)] - >>>> Failed to open zc:99@0: pfring_open error. Check if zc:99@0 exists and >>>> pf_ring module is loaded. >>>> >>>> Should I run zbalance_ipc with multiple queues? How do I specify the >>>> interfaces on >>>> the CL and the config file? FWIW I seem to get about 40% more events per >>>> second >>>> when running with multiple threads over running with 1 ZC queue. >>>> >>>> Thanks, >>>> >>>> -- >>>> Jim Hranicky >>>> Data Security Specialist >>>> UF Information Technology >>>> 105 NW 16TH ST Room #104 GAINESVILLE FL 32603-1826 >>>> 352-273-1341 >>>> _______________________________________________ >>>> Suricata IDS Users mailing list: [email protected] >>>> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/ >>>> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users >>>> Suricata User Conference November 9-11 in Washington, DC: >>>> http://oisfevents.net >>>> >>> _______________________________________________ >>> Suricata IDS Users mailing list: [email protected] >>> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/ >>> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users >>> Suricata User Conference November 9-11 in Washington, DC: >>> http://oisfevents.net >>> >> _______________________________________________ >> Suricata IDS Users mailing list: [email protected] >> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/ >> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users >> Suricata User Conference November 9-11 in Washington, DC: >> http://oisfevents.net >> > _______________________________________________ > Suricata IDS Users mailing list: [email protected] > Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/ > List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users > Suricata User Conference November 9-11 in Washington, DC: > http://oisfevents.net > _______________________________________________ > Ntop-misc mailing list > [email protected] > http://listgateway.unipi.it/mailman/listinfo/ntop-misc
signature.asc
Description: Message signed with OpenPGP using GPGMail
_______________________________________________ Ntop-misc mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop-misc
