Thanks, that's now set and now 8 start, 4 fail with bus error, 4 more start and then back to command line. /var/log/messages no longer mentions hugepages, so I guess that's not the problem any longer. The NIC's appear to go up and down repeatedly (that might be correct?), come in and out of promiscuous mode and then lots of these:
ZC[8537]: error unlink'ing /mnt/huge/pfring_zc_0: Permission denied ZC[8549]: error unlink'ing /mnt/huge/pfring_zc_1: Permission denied ZC[8568]: error unlink'ing /mnt/huge/pfring_zc_2: Permission denied etc On 4 December 2015 at 09:46, Alfredo Cardigliano <[email protected]> wrote: > node means NUMA node (i.e. CPU) > > Alfredo > > > On 04 Dec 2015, at 10:41, James <[email protected]> wrote: > > Hopefully my last stupid question - is node the same as processes and > queues in this context? So I should do that all the way up to 15? > > On 4 December 2015 at 09:32, Alfredo Cardigliano <[email protected]> > wrote: > >> >> On 04 Dec 2015, at 10:27, James <[email protected]> wrote: >> >> Thanks Alfredo, even more that you reply so quickly! I respect the "teach >> a man to fish.." method of helping, but that's a lot of parameters and >> options and I'd be making complete guesses at which ones to change and to >> what values. Would it be possible to recommend what you'd change based on >> the spec of my system? >> >> >> In essence if you have 4 nodes, you should set the number of huge pages >> per node with: >> >> $ echo 1024 > >> /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages >> $ echo 1024 > >> /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages >> $ echo 1024 > >> /sys/devices/system/node/node2/hugepages/hugepages-2048kB/nr_hugepages >> $ echo 1024 > >> /sys/devices/system/node/node3/hugepages/hugepages-2048kB/nr_hugepages >> >> I've also just noticed that the load_drive script should be changed to >> "insmod >> ./ixgbe.ko RSS=16,16" because I'm only monitoring two NIC's, is that >> correct? >> >> >> Correct >> >> Alfredo >> >> >> Thank you again >> J. >> >> On 4 December 2015 at 09:04, Alfredo Cardigliano <[email protected]> >> wrote: >> >>> Please note the total amount of pages is divided by the nodes, please >>> take a look at >>> https://github.com/ntop/PF_RING/blob/dev/doc/README.hugepages >>> >>> Alfredo >>> >>> On 04 Dec 2015, at 10:00, James <[email protected]> wrote: >>> >>> In case it helps anyone else reading this, my startup script needed some >>> corrections, ending up with: >>> >>> for i in `seq 0 1 15`; do >>> snort -q -u snort -g snort --pid-path /var/run --create-pidfile -D -c >>> /etc/snort/snort.conf -l /logs/snort/eth4_eth5/instance-$i >>> --daq-dir=/usr/local/lib/daq --daq pfring_zc --daq-mode passive -i zc:eth4@ >>> $i,zc:eth5@$i --daq-var clusterid=$i --daq-var bindcpu=$i >>> done >>> >>> i.e. I needed to remove idsbridge=1 (might be a mistake that the IDS >>> with multiqueue example does have this set in README.1st I linked earlier?) >>> and I needed to change my variable to be $i instead of $1. >>> >>> However when I run this script it only start's 4 "daemon child", then >>> gives a "bus error" on the next 4 and then returns to the command line with >>> no mention of the other 8. /var/log/messages tells me: >>> snort[4888]: FATAL ERROR: Can't initialize DAQ pfring_zc (-1) - >>> pfring_zc_daq_initialize: Cluster failed: No buffer space available (error >>> 105) >>> ZC[4897]: error mmap'ing hugepage /mnt/huge/pfring_zc_14: Cannot >>> allocate memory >>> ZC[4897]: error mmap'ing 128 hugepages of 2048 KB >>> >>> If it's relevant, when I run the ZC load_driver script (I took the >>> MQ=1,1,1,1 >>> out as advised, so that just has "insmod ./ixgbe.ko RSS=16,16,16,16") that >>> says: >>> Warning: 512 hugepages available, 1024 requested >>> >>> Things I have checked are: >>> sudo more /sys/kernel/mm/transparent_hugepage/enabled >>> always madvise [never] >>> >>> sudo more /proc/sys/vm/nr_hugepages >>> 1024 >>> >>> sudo more /proc/meminfo >>> MemTotal: 32748700 kB >>> MemFree: 27563604 kB >>> <snip> >>> AnonHugePages: 0 kB >>> HugePages_Total: 1024 >>> HugePages_Free: 1024 >>> HugePages_Rsvd: 0 >>> HugePages_Surp: 0 >>> Hugepagesize: 2048 kB >>> >>> Any ideas on how I can fix this please? >>> >>> Thanks >>> J. >>> >>> On 2 December 2015 at 15:10, Alfredo Cardigliano <[email protected]> >>> wrote: >>> >>>> >>>> On 02 Dec 2015, at 16:08, James <[email protected]> wrote: >>>> >>>> Ah.. so if only 16 queues, I should go back to only 16 copies of snort? >>>> >>>> >>>> Yes, or you can consider using zbalance_ipc for load balance in >>>> software to more queues, >>>> but I do not what is the performance you can reach with 48 queues, you >>>> should run some test. >>>> >>>> Alfredo >>>> >>>> >>>> On 2 December 2015 at 15:06, Alfredo Cardigliano <[email protected]> >>>> wrote: >>>> >>>>> Sorry, forgot to tell you RSS on ixgbe supports up to 16 queues. >>>>> >>>>> Alfredo >>>>> >>>>> On 02 Dec 2015, at 16:06, Alfredo Cardigliano <[email protected]> >>>>> wrote: >>>>> >>>>> It looks fine, you can omit MQ. >>>>> >>>>> Alfredo >>>>> >>>>> On 02 Dec 2015, at 16:04, James <[email protected]> wrote: >>>>> >>>>> Many thanks for the help Alfredo. So I'll crank things up to use all >>>>> CPU's and that gives me (I've converted it to a for loop): >>>>> >>>>> for i in `seq 0 1 48`; do >>>>> snort -q --pid-path /var/run --create-pidfile -D -c >>>>> /etc/snort/snort.conf -l /logs/snort/eth4_eth5/instance-$1 >>>>> --daq-dir=/usr/local/lib/daq --daq pfring_zc --daq-mode passive -i >>>>> zc:eth4@ >>>>> $1,zc:eth5@$1 --daq-var clusterid=$1 --daq-var idsbridge=1 --daq-var >>>>> bindcpu=$1 >>>>> done >>>>> >>>>> Is this the correct load_driver.sh setting to match? I'm not sure >>>>> about the MQ values? >>>>> insmod ./ixgbe.ko MQ=1,1,1,1 RSS=48,48,48,48 >>>>> >>>>> On 2 December 2015 at 14:15, Alfredo Cardigliano <[email protected] >>>>> > wrote: >>>>> >>>>>> Please use README.1st as reference. >>>>>> What you need to know: >>>>>> 1. Use --daq-var clusterid=K where K is a unique number per snort >>>>>> instance, used for resource allocation >>>>>> 2. Use --daq-var bindcpu=K where K is the core id for affinity, >>>>>> please ignore interrupts affinity with ZC >>>>>> 3. Use “,” in -i in please of “+” for interfaces aggregation, “+” is >>>>>> used for IPS/IDS-bridge mode >>>>>> 4. We usually recommend using only the CPU where the NIC is >>>>>> connected, however since snort is (likely) the bottleneck, feel free to >>>>>> use >>>>>> all the cores available, setting RSS=N,N where N is the number of cores >>>>>> and >>>>>> the number of snort instances. >>>>>> >>>>>> Alfredo >>>>>> >>>>>> On 02 Dec 2015, at 15:08, James <[email protected]> wrote: >>>>>> >>>>>> Follow-up question - should I use the cluster-id parameter? >>>>>> >>>>>> This uses it: >>>>>> >>>>>> https://svn.ntop.org/svn/ntop/trunk/attic/PF_RING/userland/snort/pfring-daq-module-zc/README.1st >>>>>> >>>>>> But this does not: >>>>>> http://www.ntop.org/pf_ring/accelerating-snort-with-pf_ring-dna/ >>>>>> >>>>>> On 2 December 2015 at 14:01, James <[email protected]> wrote: >>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> I posted a few weeks ago and have since got pf_ring with ZC working. >>>>>>> I'm now trying to decide how best to configure snort (in IDS mode). My >>>>>>> server has 4 X 12 core CPU's and two NIC's which are being fed one half >>>>>>> each of a 10Gb connection. >>>>>>> >>>>>>> I have a few key questions: >>>>>>> - Within the ixgbe zc load_drive.sh script, would the default 16 >>>>>>> queue option do, or would you choose something different: insmod >>>>>>> ./ixgbe.ko >>>>>>> MQ=1,1,1,1 RSS=16,16,16,16 >>>>>>> >>>>>>> - Assuming the choice of 16 above, should I start 16 copies of Snort >>>>>>> like this (variation on the example from ntop website)? >>>>>>> snort -q --pid-path /var/run --create-pidfile -D -c >>>>>>> /etc/snort/snort.conf -l /var/log/snort/eth4_eth5/instance-1 >>>>>>> --daq-dir=/usr/local/lib/daq --daq pfring_zc --daq-mode passive -i >>>>>>> zc:eth4@0 >>>>>>> +zc:eth5@0 --daq-var idsbridge=1 --daq-var bindcpu=0 >>>>>>> >>>>>>> The information on http://www.metaflows.com/features/pf_ring about >>>>>>> CPU affinity and interrupts has confused me somewhat. >>>>>>> >>>>>>> Thanks >>>>>>> J. >>>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Ntop-misc mailing list >>>>>> [email protected] >>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Ntop-misc mailing list >>>>>> [email protected] >>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>>>>> >>>>> >>>>> _______________________________________________ >>>>> Ntop-misc mailing list >>>>> [email protected] >>>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>>>> >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Ntop-misc mailing list >>>>> [email protected] >>>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>>>> >>>> >>>> _______________________________________________ >>>> Ntop-misc mailing list >>>> [email protected] >>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>>> >>>> >>>> >>>> _______________________________________________ >>>> Ntop-misc mailing list >>>> [email protected] >>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>>> >>> >>> _______________________________________________ >>> Ntop-misc mailing list >>> [email protected] >>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>> >>> >>> >>> _______________________________________________ >>> Ntop-misc mailing list >>> [email protected] >>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>> >> >> _______________________________________________ >> Ntop-misc mailing list >> [email protected] >> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >> >> >> >> _______________________________________________ >> Ntop-misc mailing list >> [email protected] >> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >> > > _______________________________________________ > Ntop-misc mailing list > [email protected] > http://listgateway.unipi.it/mailman/listinfo/ntop-misc > > > > _______________________________________________ > Ntop-misc mailing list > [email protected] > http://listgateway.unipi.it/mailman/listinfo/ntop-misc >
_______________________________________________ Ntop-misc mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop-misc
