Please note the total amount of pages is divided by the nodes, please take a look at https://github.com/ntop/PF_RING/blob/dev/doc/README.hugepages
Alfredo > On 04 Dec 2015, at 10:00, James <[email protected]> wrote: > > In case it helps anyone else reading this, my startup script needed some > corrections, ending up with: > > for i in `seq 0 1 15`; do > snort -q -u snort -g snort --pid-path /var/run --create-pidfile -D -c > /etc/snort/snort.conf -l /logs/snort/eth4_eth5/instance-$i > --daq-dir=/usr/local/lib/daq --daq pfring_zc --daq-mode passive -i > zc:eth4@$i,zc:eth5@$i --daq-var clusterid=$i --daq-var bindcpu=$i > done > > i.e. I needed to remove idsbridge=1 (might be a mistake that the IDS with > multiqueue example does have this set in README.1st I linked earlier?) and I > needed to change my variable to be $i instead of $1. > > However when I run this script it only start's 4 "daemon child", then gives a > "bus error" on the next 4 and then returns to the command line with no > mention of the other 8. /var/log/messages tells me: > snort[4888]: FATAL ERROR: Can't initialize DAQ pfring_zc (-1) - > pfring_zc_daq_initialize: Cluster failed: No buffer space available (error > 105) > ZC[4897]: error mmap'ing hugepage /mnt/huge/pfring_zc_14: Cannot allocate > memory > ZC[4897]: error mmap'ing 128 hugepages of 2048 KB > > If it's relevant, when I run the ZC load_driver script (I took the MQ=1,1,1,1 > out as advised, so that just has "insmod ./ixgbe.ko RSS=16,16,16,16") that > says: > Warning: 512 hugepages available, 1024 requested > > Things I have checked are: > sudo more /sys/kernel/mm/transparent_hugepage/enabled > always madvise [never] > > sudo more /proc/sys/vm/nr_hugepages > 1024 > > sudo more /proc/meminfo > MemTotal: 32748700 kB > MemFree: 27563604 kB > <snip> > AnonHugePages: 0 kB > HugePages_Total: 1024 > HugePages_Free: 1024 > HugePages_Rsvd: 0 > HugePages_Surp: 0 > Hugepagesize: 2048 kB > > Any ideas on how I can fix this please? > > Thanks > J. > > On 2 December 2015 at 15:10, Alfredo Cardigliano <[email protected] > <mailto:[email protected]>> wrote: > >> On 02 Dec 2015, at 16:08, James <[email protected] >> <mailto:[email protected]>> wrote: >> >> Ah.. so if only 16 queues, I should go back to only 16 copies of snort? > > Yes, or you can consider using zbalance_ipc for load balance in software to > more queues, > but I do not what is the performance you can reach with 48 queues, you should > run some test. > > Alfredo > >> >> On 2 December 2015 at 15:06, Alfredo Cardigliano <[email protected] >> <mailto:[email protected]>> wrote: >> Sorry, forgot to tell you RSS on ixgbe supports up to 16 queues. >> >> Alfredo >> >>> On 02 Dec 2015, at 16:06, Alfredo Cardigliano <[email protected] >>> <mailto:[email protected]>> wrote: >>> >>> It looks fine, you can omit MQ. >>> >>> Alfredo >>> >>>> On 02 Dec 2015, at 16:04, James <[email protected] >>>> <mailto:[email protected]>> wrote: >>>> >>>> Many thanks for the help Alfredo. So I'll crank things up to use all CPU's >>>> and that gives me (I've converted it to a for loop): >>>> >>>> for i in `seq 0 1 48`; do >>>> snort -q --pid-path /var/run --create-pidfile -D -c /etc/snort/snort.conf >>>> -l /logs/snort/eth4_eth5/instance-$1 --daq-dir=/usr/local/lib/daq --daq >>>> pfring_zc --daq-mode passive -i zc:eth4@$1,zc:eth5@$1 --daq-var >>>> clusterid=$1 --daq-var idsbridge=1 --daq-var bindcpu=$1 >>>> done >>>> >>>> Is this the correct load_driver.sh setting to match? I'm not sure about >>>> the MQ values? >>>> insmod ./ixgbe.ko MQ=1,1,1,1 RSS=48,48,48,48 >>>> >>>> On 2 December 2015 at 14:15, Alfredo Cardigliano <[email protected] >>>> <mailto:[email protected]>> wrote: >>>> Please use README.1st as reference. >>>> What you need to know: >>>> 1. Use --daq-var clusterid=K where K is a unique number per snort >>>> instance, used for resource allocation >>>> 2. Use --daq-var bindcpu=K where K is the core id for affinity, please >>>> ignore interrupts affinity with ZC >>>> 3. Use “,” in -i in please of “+” for interfaces aggregation, “+” is used >>>> for IPS/IDS-bridge mode >>>> 4. We usually recommend using only the CPU where the NIC is connected, >>>> however since snort is (likely) the bottleneck, feel free to use all the >>>> cores available, setting RSS=N,N where N is the number of cores and the >>>> number of snort instances. >>>> >>>> Alfredo >>>> >>>>> On 02 Dec 2015, at 15:08, James <[email protected] >>>>> <mailto:[email protected]>> wrote: >>>>> >>>>> Follow-up question - should I use the cluster-id parameter? >>>>> >>>>> This uses it: >>>>> https://svn.ntop.org/svn/ntop/trunk/attic/PF_RING/userland/snort/pfring-daq-module-zc/README.1st >>>>> >>>>> <https://svn.ntop.org/svn/ntop/trunk/attic/PF_RING/userland/snort/pfring-daq-module-zc/README.1st> >>>>> >>>>> But this does not: >>>>> http://www.ntop.org/pf_ring/accelerating-snort-with-pf_ring-dna/ >>>>> <http://www.ntop.org/pf_ring/accelerating-snort-with-pf_ring-dna/> >>>>> >>>>> On 2 December 2015 at 14:01, James <[email protected] >>>>> <mailto:[email protected]>> wrote: >>>>> Hi all, >>>>> >>>>> I posted a few weeks ago and have since got pf_ring with ZC working. I'm >>>>> now trying to decide how best to configure snort (in IDS mode). My server >>>>> has 4 X 12 core CPU's and two NIC's which are being fed one half each of >>>>> a 10Gb connection. >>>>> >>>>> I have a few key questions: >>>>> - Within the ixgbe zc load_drive.sh script, would the default 16 queue >>>>> option do, or would you choose something different: insmod ./ixgbe.ko >>>>> MQ=1,1,1,1 RSS=16,16,16,16 >>>>> >>>>> - Assuming the choice of 16 above, should I start 16 copies of Snort like >>>>> this (variation on the example from ntop website)? >>>>> snort -q --pid-path /var/run --create-pidfile -D -c /etc/snort/snort.conf >>>>> -l /var/log/snort/eth4_eth5/instance-1 --daq-dir=/usr/local/lib/daq --daq >>>>> pfring_zc --daq-mode passive -i zc:eth4@0+zc:eth5@0 --daq-var idsbridge=1 >>>>> --daq-var bindcpu=0 >>>>> >>>>> The information on http://www.metaflows.com/features/pf_ring >>>>> <http://www.metaflows.com/features/pf_ring> about CPU affinity and >>>>> interrupts has confused me somewhat. >>>>> >>>>> Thanks >>>>> J. >>>>> >>>>> _______________________________________________ >>>>> Ntop-misc mailing list >>>>> [email protected] <mailto:[email protected]> >>>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>>>> <http://listgateway.unipi.it/mailman/listinfo/ntop-misc> >>>> >>>> _______________________________________________ >>>> Ntop-misc mailing list >>>> [email protected] <mailto:[email protected]> >>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>>> <http://listgateway.unipi.it/mailman/listinfo/ntop-misc> >>>> >>>> _______________________________________________ >>>> Ntop-misc mailing list >>>> [email protected] <mailto:[email protected]> >>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>>> <http://listgateway.unipi.it/mailman/listinfo/ntop-misc> >> >> >> _______________________________________________ >> Ntop-misc mailing list >> [email protected] <mailto:[email protected]> >> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >> <http://listgateway.unipi.it/mailman/listinfo/ntop-misc> >> >> _______________________________________________ >> Ntop-misc mailing list >> [email protected] <mailto:[email protected]> >> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >> <http://listgateway.unipi.it/mailman/listinfo/ntop-misc> > > _______________________________________________ > Ntop-misc mailing list > [email protected] <mailto:[email protected]> > http://listgateway.unipi.it/mailman/listinfo/ntop-misc > <http://listgateway.unipi.it/mailman/listinfo/ntop-misc> > > _______________________________________________ > Ntop-misc mailing list > [email protected] > http://listgateway.unipi.it/mailman/listinfo/ntop-misc
_______________________________________________ Ntop-misc mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop-misc
