I'm another step forward now, thank you Alfredo. I can get 12 instances to start (for some reason it's always instances 0-7 and 12-15). Instances 8-11 give the same "bus error" at startup though. There are no errors in /var/log/messages
On 4 December 2015 at 11:03, James <[email protected]> wrote: > Thanks, that's now set and now 8 start, 4 fail with bus error, 4 more > start and then back to command line. /var/log/messages no longer mentions > hugepages, so I guess that's not the problem any longer. The NIC's appear > to go up and down repeatedly (that might be correct?), come in and out of > promiscuous mode and then lots of these: > > ZC[8537]: error unlink'ing /mnt/huge/pfring_zc_0: Permission denied > ZC[8549]: error unlink'ing /mnt/huge/pfring_zc_1: Permission denied > ZC[8568]: error unlink'ing /mnt/huge/pfring_zc_2: Permission denied > etc > > > On 4 December 2015 at 09:46, Alfredo Cardigliano <[email protected]> > wrote: > >> node means NUMA node (i.e. CPU) >> >> Alfredo >> >> >> On 04 Dec 2015, at 10:41, James <[email protected]> wrote: >> >> Hopefully my last stupid question - is node the same as processes and >> queues in this context? So I should do that all the way up to 15? >> >> On 4 December 2015 at 09:32, Alfredo Cardigliano <[email protected]> >> wrote: >> >>> >>> On 04 Dec 2015, at 10:27, James <[email protected]> wrote: >>> >>> Thanks Alfredo, even more that you reply so quickly! I respect the >>> "teach a man to fish.." method of helping, but that's a lot of parameters >>> and options and I'd be making complete guesses at which ones to change and >>> to what values. Would it be possible to recommend what you'd change based >>> on the spec of my system? >>> >>> >>> In essence if you have 4 nodes, you should set the number of huge pages >>> per node with: >>> >>> $ echo 1024 > >>> /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages >>> $ echo 1024 > >>> /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages >>> $ echo 1024 > >>> /sys/devices/system/node/node2/hugepages/hugepages-2048kB/nr_hugepages >>> $ echo 1024 > >>> /sys/devices/system/node/node3/hugepages/hugepages-2048kB/nr_hugepages >>> >>> I've also just noticed that the load_drive script should be changed to >>> "insmod >>> ./ixgbe.ko RSS=16,16" because I'm only monitoring two NIC's, is that >>> correct? >>> >>> >>> Correct >>> >>> Alfredo >>> >>> >>> Thank you again >>> J. >>> >>> On 4 December 2015 at 09:04, Alfredo Cardigliano <[email protected]> >>> wrote: >>> >>>> Please note the total amount of pages is divided by the nodes, please >>>> take a look at >>>> https://github.com/ntop/PF_RING/blob/dev/doc/README.hugepages >>>> >>>> Alfredo >>>> >>>> On 04 Dec 2015, at 10:00, James <[email protected]> wrote: >>>> >>>> In case it helps anyone else reading this, my startup script needed >>>> some corrections, ending up with: >>>> >>>> for i in `seq 0 1 15`; do >>>> snort -q -u snort -g snort --pid-path /var/run --create-pidfile -D -c >>>> /etc/snort/snort.conf -l /logs/snort/eth4_eth5/instance-$i >>>> --daq-dir=/usr/local/lib/daq --daq pfring_zc --daq-mode passive -i zc:eth4@ >>>> $i,zc:eth5@$i --daq-var clusterid=$i --daq-var bindcpu=$i >>>> done >>>> >>>> i.e. I needed to remove idsbridge=1 (might be a mistake that the IDS >>>> with multiqueue example does have this set in README.1st I linked earlier?) >>>> and I needed to change my variable to be $i instead of $1. >>>> >>>> However when I run this script it only start's 4 "daemon child", then >>>> gives a "bus error" on the next 4 and then returns to the command line with >>>> no mention of the other 8. /var/log/messages tells me: >>>> snort[4888]: FATAL ERROR: Can't initialize DAQ pfring_zc (-1) - >>>> pfring_zc_daq_initialize: Cluster failed: No buffer space available (error >>>> 105) >>>> ZC[4897]: error mmap'ing hugepage /mnt/huge/pfring_zc_14: Cannot >>>> allocate memory >>>> ZC[4897]: error mmap'ing 128 hugepages of 2048 KB >>>> >>>> If it's relevant, when I run the ZC load_driver script (I took the >>>> MQ=1,1,1,1 >>>> out as advised, so that just has "insmod ./ixgbe.ko RSS=16,16,16,16") that >>>> says: >>>> Warning: 512 hugepages available, 1024 requested >>>> >>>> Things I have checked are: >>>> sudo more /sys/kernel/mm/transparent_hugepage/enabled >>>> always madvise [never] >>>> >>>> sudo more /proc/sys/vm/nr_hugepages >>>> 1024 >>>> >>>> sudo more /proc/meminfo >>>> MemTotal: 32748700 kB >>>> MemFree: 27563604 kB >>>> <snip> >>>> AnonHugePages: 0 kB >>>> HugePages_Total: 1024 >>>> HugePages_Free: 1024 >>>> HugePages_Rsvd: 0 >>>> HugePages_Surp: 0 >>>> Hugepagesize: 2048 kB >>>> >>>> Any ideas on how I can fix this please? >>>> >>>> Thanks >>>> J. >>>> >>>> On 2 December 2015 at 15:10, Alfredo Cardigliano <[email protected]> >>>> wrote: >>>> >>>>> >>>>> On 02 Dec 2015, at 16:08, James <[email protected]> wrote: >>>>> >>>>> Ah.. so if only 16 queues, I should go back to only 16 copies of snort? >>>>> >>>>> >>>>> Yes, or you can consider using zbalance_ipc for load balance in >>>>> software to more queues, >>>>> but I do not what is the performance you can reach with 48 queues, you >>>>> should run some test. >>>>> >>>>> Alfredo >>>>> >>>>> >>>>> On 2 December 2015 at 15:06, Alfredo Cardigliano <[email protected] >>>>> > wrote: >>>>> >>>>>> Sorry, forgot to tell you RSS on ixgbe supports up to 16 queues. >>>>>> >>>>>> Alfredo >>>>>> >>>>>> On 02 Dec 2015, at 16:06, Alfredo Cardigliano <[email protected]> >>>>>> wrote: >>>>>> >>>>>> It looks fine, you can omit MQ. >>>>>> >>>>>> Alfredo >>>>>> >>>>>> On 02 Dec 2015, at 16:04, James <[email protected]> wrote: >>>>>> >>>>>> Many thanks for the help Alfredo. So I'll crank things up to use all >>>>>> CPU's and that gives me (I've converted it to a for loop): >>>>>> >>>>>> for i in `seq 0 1 48`; do >>>>>> snort -q --pid-path /var/run --create-pidfile -D -c >>>>>> /etc/snort/snort.conf -l /logs/snort/eth4_eth5/instance-$1 >>>>>> --daq-dir=/usr/local/lib/daq --daq pfring_zc --daq-mode passive -i >>>>>> zc:eth4@ >>>>>> $1,zc:eth5@$1 --daq-var clusterid=$1 --daq-var idsbridge=1 --daq-var >>>>>> bindcpu=$1 >>>>>> done >>>>>> >>>>>> Is this the correct load_driver.sh setting to match? I'm not sure >>>>>> about the MQ values? >>>>>> insmod ./ixgbe.ko MQ=1,1,1,1 RSS=48,48,48,48 >>>>>> >>>>>> On 2 December 2015 at 14:15, Alfredo Cardigliano < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Please use README.1st as reference. >>>>>>> What you need to know: >>>>>>> 1. Use --daq-var clusterid=K where K is a unique number per snort >>>>>>> instance, used for resource allocation >>>>>>> 2. Use --daq-var bindcpu=K where K is the core id for affinity, >>>>>>> please ignore interrupts affinity with ZC >>>>>>> 3. Use “,” in -i in please of “+” for interfaces aggregation, “+” is >>>>>>> used for IPS/IDS-bridge mode >>>>>>> 4. We usually recommend using only the CPU where the NIC is >>>>>>> connected, however since snort is (likely) the bottleneck, feel free to >>>>>>> use >>>>>>> all the cores available, setting RSS=N,N where N is the number of cores >>>>>>> and >>>>>>> the number of snort instances. >>>>>>> >>>>>>> Alfredo >>>>>>> >>>>>>> On 02 Dec 2015, at 15:08, James <[email protected]> wrote: >>>>>>> >>>>>>> Follow-up question - should I use the cluster-id parameter? >>>>>>> >>>>>>> This uses it: >>>>>>> >>>>>>> https://svn.ntop.org/svn/ntop/trunk/attic/PF_RING/userland/snort/pfring-daq-module-zc/README.1st >>>>>>> >>>>>>> But this does not: >>>>>>> http://www.ntop.org/pf_ring/accelerating-snort-with-pf_ring-dna/ >>>>>>> >>>>>>> On 2 December 2015 at 14:01, James <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi all, >>>>>>>> >>>>>>>> I posted a few weeks ago and have since got pf_ring with ZC >>>>>>>> working. I'm now trying to decide how best to configure snort (in IDS >>>>>>>> mode). My server has 4 X 12 core CPU's and two NIC's which are being >>>>>>>> fed >>>>>>>> one half each of a 10Gb connection. >>>>>>>> >>>>>>>> I have a few key questions: >>>>>>>> - Within the ixgbe zc load_drive.sh script, would the default 16 >>>>>>>> queue option do, or would you choose something different: insmod >>>>>>>> ./ixgbe.ko >>>>>>>> MQ=1,1,1,1 RSS=16,16,16,16 >>>>>>>> >>>>>>>> - Assuming the choice of 16 above, should I start 16 copies of >>>>>>>> Snort like this (variation on the example from ntop website)? >>>>>>>> snort -q --pid-path /var/run --create-pidfile -D -c >>>>>>>> /etc/snort/snort.conf -l /var/log/snort/eth4_eth5/instance-1 >>>>>>>> --daq-dir=/usr/local/lib/daq --daq pfring_zc --daq-mode passive -i >>>>>>>> zc:eth4@0 >>>>>>>> +zc:eth5@0 --daq-var idsbridge=1 --daq-var bindcpu=0 >>>>>>>> >>>>>>>> The information on http://www.metaflows.com/features/pf_ring about >>>>>>>> CPU affinity and interrupts has confused me somewhat. >>>>>>>> >>>>>>>> Thanks >>>>>>>> J. >>>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Ntop-misc mailing list >>>>>>> [email protected] >>>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Ntop-misc mailing list >>>>>>> [email protected] >>>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Ntop-misc mailing list >>>>>> [email protected] >>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Ntop-misc mailing list >>>>>> [email protected] >>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>>>>> >>>>> >>>>> _______________________________________________ >>>>> Ntop-misc mailing list >>>>> [email protected] >>>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Ntop-misc mailing list >>>>> [email protected] >>>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>>>> >>>> >>>> _______________________________________________ >>>> Ntop-misc mailing list >>>> [email protected] >>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>>> >>>> >>>> >>>> _______________________________________________ >>>> Ntop-misc mailing list >>>> [email protected] >>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>>> >>> >>> _______________________________________________ >>> Ntop-misc mailing list >>> [email protected] >>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>> >>> >>> >>> _______________________________________________ >>> Ntop-misc mailing list >>> [email protected] >>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>> >> >> _______________________________________________ >> Ntop-misc mailing list >> [email protected] >> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >> >> >> >> _______________________________________________ >> Ntop-misc mailing list >> [email protected] >> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >> > >
_______________________________________________ Ntop-misc mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop-misc
