> On 04 Dec 2015, at 10:27, James <[email protected]> wrote: > > Thanks Alfredo, even more that you reply so quickly! I respect the "teach a > man to fish.." method of helping, but that's a lot of parameters and options > and I'd be making complete guesses at which ones to change and to what > values. Would it be possible to recommend what you'd change based on the spec > of my system?
In essence if you have 4 nodes, you should set the number of huge pages per node with: $ echo 1024 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages $ echo 1024 > /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages $ echo 1024 > /sys/devices/system/node/node2/hugepages/hugepages-2048kB/nr_hugepages $ echo 1024 > /sys/devices/system/node/node3/hugepages/hugepages-2048kB/nr_hugepages > I've also just noticed that the load_drive script should be changed to > "insmod ./ixgbe.ko RSS=16,16" because I'm only monitoring two NIC's, is that > correct? Correct Alfredo > > Thank you again > J. > > On 4 December 2015 at 09:04, Alfredo Cardigliano <[email protected] > <mailto:[email protected]>> wrote: > Please note the total amount of pages is divided by the nodes, please take a > look at https://github.com/ntop/PF_RING/blob/dev/doc/README.hugepages > <https://github.com/ntop/PF_RING/blob/dev/doc/README.hugepages> > > Alfredo > >> On 04 Dec 2015, at 10:00, James <[email protected] >> <mailto:[email protected]>> wrote: >> >> In case it helps anyone else reading this, my startup script needed some >> corrections, ending up with: >> >> for i in `seq 0 1 15`; do >> snort -q -u snort -g snort --pid-path /var/run --create-pidfile -D -c >> /etc/snort/snort.conf -l /logs/snort/eth4_eth5/instance-$i >> --daq-dir=/usr/local/lib/daq --daq pfring_zc --daq-mode passive -i >> zc:eth4@$i,zc:eth5@$i --daq-var clusterid=$i --daq-var bindcpu=$i >> done >> >> i.e. I needed to remove idsbridge=1 (might be a mistake that the IDS with >> multiqueue example does have this set in README.1st I linked earlier?) and I >> needed to change my variable to be $i instead of $1. >> >> However when I run this script it only start's 4 "daemon child", then gives >> a "bus error" on the next 4 and then returns to the command line with no >> mention of the other 8. /var/log/messages tells me: >> snort[4888]: FATAL ERROR: Can't initialize DAQ pfring_zc (-1) - >> pfring_zc_daq_initialize: Cluster failed: No buffer space available (error >> 105) >> ZC[4897]: error mmap'ing hugepage /mnt/huge/pfring_zc_14: Cannot allocate >> memory >> ZC[4897]: error mmap'ing 128 hugepages of 2048 KB >> >> If it's relevant, when I run the ZC load_driver script (I took the >> MQ=1,1,1,1 out as advised, so that just has "insmod ./ixgbe.ko >> RSS=16,16,16,16") that says: >> Warning: 512 hugepages available, 1024 requested >> >> Things I have checked are: >> sudo more /sys/kernel/mm/transparent_hugepage/enabled >> always madvise [never] >> >> sudo more /proc/sys/vm/nr_hugepages >> 1024 >> >> sudo more /proc/meminfo >> MemTotal: 32748700 kB >> MemFree: 27563604 kB >> <snip> >> AnonHugePages: 0 kB >> HugePages_Total: 1024 >> HugePages_Free: 1024 >> HugePages_Rsvd: 0 >> HugePages_Surp: 0 >> Hugepagesize: 2048 kB >> >> Any ideas on how I can fix this please? >> >> Thanks >> J. >> >> On 2 December 2015 at 15:10, Alfredo Cardigliano <[email protected] >> <mailto:[email protected]>> wrote: >> >>> On 02 Dec 2015, at 16:08, James <[email protected] >>> <mailto:[email protected]>> wrote: >>> >>> Ah.. so if only 16 queues, I should go back to only 16 copies of snort? >> >> Yes, or you can consider using zbalance_ipc for load balance in software to >> more queues, >> but I do not what is the performance you can reach with 48 queues, you >> should run some test. >> >> Alfredo >> >>> >>> On 2 December 2015 at 15:06, Alfredo Cardigliano <[email protected] >>> <mailto:[email protected]>> wrote: >>> Sorry, forgot to tell you RSS on ixgbe supports up to 16 queues. >>> >>> Alfredo >>> >>>> On 02 Dec 2015, at 16:06, Alfredo Cardigliano <[email protected] >>>> <mailto:[email protected]>> wrote: >>>> >>>> It looks fine, you can omit MQ. >>>> >>>> Alfredo >>>> >>>>> On 02 Dec 2015, at 16:04, James <[email protected] >>>>> <mailto:[email protected]>> wrote: >>>>> >>>>> Many thanks for the help Alfredo. So I'll crank things up to use all >>>>> CPU's and that gives me (I've converted it to a for loop): >>>>> >>>>> for i in `seq 0 1 48`; do >>>>> snort -q --pid-path /var/run --create-pidfile -D -c /etc/snort/snort.conf >>>>> -l /logs/snort/eth4_eth5/instance-$1 --daq-dir=/usr/local/lib/daq --daq >>>>> pfring_zc --daq-mode passive -i zc:eth4@$1,zc:eth5@$1 --daq-var >>>>> clusterid=$1 --daq-var idsbridge=1 --daq-var bindcpu=$1 >>>>> done >>>>> >>>>> Is this the correct load_driver.sh setting to match? I'm not sure about >>>>> the MQ values? >>>>> insmod ./ixgbe.ko MQ=1,1,1,1 RSS=48,48,48,48 >>>>> >>>>> On 2 December 2015 at 14:15, Alfredo Cardigliano <[email protected] >>>>> <mailto:[email protected]>> wrote: >>>>> Please use README.1st as reference. >>>>> What you need to know: >>>>> 1. Use --daq-var clusterid=K where K is a unique number per snort >>>>> instance, used for resource allocation >>>>> 2. Use --daq-var bindcpu=K where K is the core id for affinity, please >>>>> ignore interrupts affinity with ZC >>>>> 3. Use “,” in -i in please of “+” for interfaces aggregation, “+” is used >>>>> for IPS/IDS-bridge mode >>>>> 4. We usually recommend using only the CPU where the NIC is connected, >>>>> however since snort is (likely) the bottleneck, feel free to use all the >>>>> cores available, setting RSS=N,N where N is the number of cores and the >>>>> number of snort instances. >>>>> >>>>> Alfredo >>>>> >>>>>> On 02 Dec 2015, at 15:08, James <[email protected] >>>>>> <mailto:[email protected]>> wrote: >>>>>> >>>>>> Follow-up question - should I use the cluster-id parameter? >>>>>> >>>>>> This uses it: >>>>>> https://svn.ntop.org/svn/ntop/trunk/attic/PF_RING/userland/snort/pfring-daq-module-zc/README.1st >>>>>> >>>>>> <https://svn.ntop.org/svn/ntop/trunk/attic/PF_RING/userland/snort/pfring-daq-module-zc/README.1st> >>>>>> >>>>>> But this does not: >>>>>> http://www.ntop.org/pf_ring/accelerating-snort-with-pf_ring-dna/ >>>>>> <http://www.ntop.org/pf_ring/accelerating-snort-with-pf_ring-dna/> >>>>>> >>>>>> On 2 December 2015 at 14:01, James <[email protected] >>>>>> <mailto:[email protected]>> wrote: >>>>>> Hi all, >>>>>> >>>>>> I posted a few weeks ago and have since got pf_ring with ZC working. I'm >>>>>> now trying to decide how best to configure snort (in IDS mode). My >>>>>> server has 4 X 12 core CPU's and two NIC's which are being fed one half >>>>>> each of a 10Gb connection. >>>>>> >>>>>> I have a few key questions: >>>>>> - Within the ixgbe zc load_drive.sh script, would the default 16 queue >>>>>> option do, or would you choose something different: insmod ./ixgbe.ko >>>>>> MQ=1,1,1,1 RSS=16,16,16,16 >>>>>> >>>>>> - Assuming the choice of 16 above, should I start 16 copies of Snort >>>>>> like this (variation on the example from ntop website)? >>>>>> snort -q --pid-path /var/run --create-pidfile -D -c >>>>>> /etc/snort/snort.conf -l /var/log/snort/eth4_eth5/instance-1 >>>>>> --daq-dir=/usr/local/lib/daq --daq pfring_zc --daq-mode passive -i >>>>>> zc:eth4@0+zc:eth5@0 --daq-var idsbridge=1 --daq-var bindcpu=0 >>>>>> >>>>>> The information on http://www.metaflows.com/features/pf_ring >>>>>> <http://www.metaflows.com/features/pf_ring> about CPU affinity and >>>>>> interrupts has confused me somewhat. >>>>>> >>>>>> Thanks >>>>>> J. >>>>>> >>>>>> _______________________________________________ >>>>>> Ntop-misc mailing list >>>>>> [email protected] <mailto:[email protected]> >>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>>>>> <http://listgateway.unipi.it/mailman/listinfo/ntop-misc> >>>>> >>>>> _______________________________________________ >>>>> Ntop-misc mailing list >>>>> [email protected] <mailto:[email protected]> >>>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>>>> <http://listgateway.unipi.it/mailman/listinfo/ntop-misc> >>>>> >>>>> _______________________________________________ >>>>> Ntop-misc mailing list >>>>> [email protected] <mailto:[email protected]> >>>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>>>> <http://listgateway.unipi.it/mailman/listinfo/ntop-misc> >>> >>> >>> _______________________________________________ >>> Ntop-misc mailing list >>> [email protected] <mailto:[email protected]> >>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>> <http://listgateway.unipi.it/mailman/listinfo/ntop-misc> >>> >>> _______________________________________________ >>> Ntop-misc mailing list >>> [email protected] <mailto:[email protected]> >>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>> <http://listgateway.unipi.it/mailman/listinfo/ntop-misc> >> >> _______________________________________________ >> Ntop-misc mailing list >> [email protected] <mailto:[email protected]> >> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >> <http://listgateway.unipi.it/mailman/listinfo/ntop-misc> >> >> _______________________________________________ >> Ntop-misc mailing list >> [email protected] <mailto:[email protected]> >> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >> <http://listgateway.unipi.it/mailman/listinfo/ntop-misc> > > _______________________________________________ > Ntop-misc mailing list > [email protected] <mailto:[email protected]> > http://listgateway.unipi.it/mailman/listinfo/ntop-misc > <http://listgateway.unipi.it/mailman/listinfo/ntop-misc> > > _______________________________________________ > Ntop-misc mailing list > [email protected] > http://listgateway.unipi.it/mailman/listinfo/ntop-misc
_______________________________________________ Ntop-misc mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop-misc
