Hi James for debug purposes, do you see any improvement if you change the number of ring slots from 32k to 4k for instance? (i.e. ethtool -G ethX rx 4096)
Alfredo > On 07 Dec 2015, at 15:15, James <[email protected]> wrote: > > I'm another step forward now, thank you Alfredo. I can get 12 instances to > start (for some reason it's always instances 0-7 and 12-15). Instances 8-11 > give the same "bus error" at startup though. There are no errors in > /var/log/messages > > > > On 4 December 2015 at 11:03, James <[email protected] > <mailto:[email protected]>> wrote: > Thanks, that's now set and now 8 start, 4 fail with bus error, 4 more start > and then back to command line. /var/log/messages no longer mentions > hugepages, so I guess that's not the problem any longer. The NIC's appear to > go up and down repeatedly (that might be correct?), come in and out of > promiscuous mode and then lots of these: > > ZC[8537]: error unlink'ing /mnt/huge/pfring_zc_0: Permission denied > ZC[8549]: error unlink'ing /mnt/huge/pfring_zc_1: Permission denied > ZC[8568]: error unlink'ing /mnt/huge/pfring_zc_2: Permission denied > etc > > > On 4 December 2015 at 09:46, Alfredo Cardigliano <[email protected] > <mailto:[email protected]>> wrote: > node means NUMA node (i.e. CPU) > > Alfredo > > >> On 04 Dec 2015, at 10:41, James <[email protected] >> <mailto:[email protected]>> wrote: >> >> Hopefully my last stupid question - is node the same as processes and queues >> in this context? So I should do that all the way up to 15? >> >> On 4 December 2015 at 09:32, Alfredo Cardigliano <[email protected] >> <mailto:[email protected]>> wrote: >> >>> On 04 Dec 2015, at 10:27, James <[email protected] >>> <mailto:[email protected]>> wrote: >>> >>> Thanks Alfredo, even more that you reply so quickly! I respect the "teach a >>> man to fish.." method of helping, but that's a lot of parameters and >>> options and I'd be making complete guesses at which ones to change and to >>> what values. Would it be possible to recommend what you'd change based on >>> the spec of my system? >> >> In essence if you have 4 nodes, you should set the number of huge pages per >> node with: >> >> $ echo 1024 > >> /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages >> $ echo 1024 > >> /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages >> $ echo 1024 > >> /sys/devices/system/node/node2/hugepages/hugepages-2048kB/nr_hugepages >> $ echo 1024 > >> /sys/devices/system/node/node3/hugepages/hugepages-2048kB/nr_hugepages >> >>> I've also just noticed that the load_drive script should be changed to >>> "insmod ./ixgbe.ko RSS=16,16" because I'm only monitoring two NIC's, is >>> that correct? >> >> Correct >> >> Alfredo >> >>> >>> Thank you again >>> J. >>> >>> On 4 December 2015 at 09:04, Alfredo Cardigliano <[email protected] >>> <mailto:[email protected]>> wrote: >>> Please note the total amount of pages is divided by the nodes, please take >>> a look at https://github.com/ntop/PF_RING/blob/dev/doc/README.hugepages >>> <https://github.com/ntop/PF_RING/blob/dev/doc/README.hugepages> >>> >>> Alfredo >>> >>>> On 04 Dec 2015, at 10:00, James <[email protected] >>>> <mailto:[email protected]>> wrote: >>>> >>>> In case it helps anyone else reading this, my startup script needed some >>>> corrections, ending up with: >>>> >>>> for i in `seq 0 1 15`; do >>>> snort -q -u snort -g snort --pid-path /var/run --create-pidfile -D -c >>>> /etc/snort/snort.conf -l /logs/snort/eth4_eth5/instance-$i >>>> --daq-dir=/usr/local/lib/daq --daq pfring_zc --daq-mode passive -i >>>> zc:eth4@$i,zc:eth5@$i --daq-var clusterid=$i --daq-var bindcpu=$i >>>> done >>>> >>>> i.e. I needed to remove idsbridge=1 (might be a mistake that the IDS with >>>> multiqueue example does have this set in README.1st I linked earlier?) and >>>> I needed to change my variable to be $i instead of $1. >>>> >>>> However when I run this script it only start's 4 "daemon child", then >>>> gives a "bus error" on the next 4 and then returns to the command line >>>> with no mention of the other 8. /var/log/messages tells me: >>>> snort[4888]: FATAL ERROR: Can't initialize DAQ pfring_zc (-1) - >>>> pfring_zc_daq_initialize: Cluster failed: No buffer space available (error >>>> 105) >>>> ZC[4897]: error mmap'ing hugepage /mnt/huge/pfring_zc_14: Cannot allocate >>>> memory >>>> ZC[4897]: error mmap'ing 128 hugepages of 2048 KB >>>> >>>> If it's relevant, when I run the ZC load_driver script (I took the >>>> MQ=1,1,1,1 out as advised, so that just has "insmod ./ixgbe.ko >>>> RSS=16,16,16,16") that says: >>>> Warning: 512 hugepages available, 1024 requested >>>> >>>> Things I have checked are: >>>> sudo more /sys/kernel/mm/transparent_hugepage/enabled >>>> always madvise [never] >>>> >>>> sudo more /proc/sys/vm/nr_hugepages >>>> 1024 >>>> >>>> sudo more /proc/meminfo >>>> MemTotal: 32748700 kB >>>> MemFree: 27563604 kB >>>> <snip> >>>> AnonHugePages: 0 kB >>>> HugePages_Total: 1024 >>>> HugePages_Free: 1024 >>>> HugePages_Rsvd: 0 >>>> HugePages_Surp: 0 >>>> Hugepagesize: 2048 kB >>>> >>>> Any ideas on how I can fix this please? >>>> >>>> Thanks >>>> J. >>>> >>>> On 2 December 2015 at 15:10, Alfredo Cardigliano <[email protected] >>>> <mailto:[email protected]>> wrote: >>>> >>>>> On 02 Dec 2015, at 16:08, James <[email protected] >>>>> <mailto:[email protected]>> wrote: >>>>> >>>>> Ah.. so if only 16 queues, I should go back to only 16 copies of snort? >>>> >>>> Yes, or you can consider using zbalance_ipc for load balance in software >>>> to more queues, >>>> but I do not what is the performance you can reach with 48 queues, you >>>> should run some test. >>>> >>>> Alfredo >>>> >>>>> >>>>> On 2 December 2015 at 15:06, Alfredo Cardigliano <[email protected] >>>>> <mailto:[email protected]>> wrote: >>>>> Sorry, forgot to tell you RSS on ixgbe supports up to 16 queues. >>>>> >>>>> Alfredo >>>>> >>>>>> On 02 Dec 2015, at 16:06, Alfredo Cardigliano <[email protected] >>>>>> <mailto:[email protected]>> wrote: >>>>>> >>>>>> It looks fine, you can omit MQ. >>>>>> >>>>>> Alfredo >>>>>> >>>>>>> On 02 Dec 2015, at 16:04, James <[email protected] >>>>>>> <mailto:[email protected]>> wrote: >>>>>>> >>>>>>> Many thanks for the help Alfredo. So I'll crank things up to use all >>>>>>> CPU's and that gives me (I've converted it to a for loop): >>>>>>> >>>>>>> for i in `seq 0 1 48`; do >>>>>>> snort -q --pid-path /var/run --create-pidfile -D -c >>>>>>> /etc/snort/snort.conf -l /logs/snort/eth4_eth5/instance-$1 >>>>>>> --daq-dir=/usr/local/lib/daq --daq pfring_zc --daq-mode passive -i >>>>>>> zc:eth4@$1,zc:eth5@$1 --daq-var clusterid=$1 --daq-var idsbridge=1 >>>>>>> --daq-var bindcpu=$1 >>>>>>> done >>>>>>> >>>>>>> Is this the correct load_driver.sh setting to match? I'm not sure about >>>>>>> the MQ values? >>>>>>> insmod ./ixgbe.ko MQ=1,1,1,1 RSS=48,48,48,48 >>>>>>> >>>>>>> On 2 December 2015 at 14:15, Alfredo Cardigliano <[email protected] >>>>>>> <mailto:[email protected]>> wrote: >>>>>>> Please use README.1st as reference. >>>>>>> What you need to know: >>>>>>> 1. Use --daq-var clusterid=K where K is a unique number per snort >>>>>>> instance, used for resource allocation >>>>>>> 2. Use --daq-var bindcpu=K where K is the core id for affinity, please >>>>>>> ignore interrupts affinity with ZC >>>>>>> 3. Use “,” in -i in please of “+” for interfaces aggregation, “+” is >>>>>>> used for IPS/IDS-bridge mode >>>>>>> 4. We usually recommend using only the CPU where the NIC is connected, >>>>>>> however since snort is (likely) the bottleneck, feel free to use all >>>>>>> the cores available, setting RSS=N,N where N is the number of cores and >>>>>>> the number of snort instances. >>>>>>> >>>>>>> Alfredo >>>>>>> >>>>>>>> On 02 Dec 2015, at 15:08, James <[email protected] >>>>>>>> <mailto:[email protected]>> wrote: >>>>>>>> >>>>>>>> Follow-up question - should I use the cluster-id parameter? >>>>>>>> >>>>>>>> This uses it: >>>>>>>> https://svn.ntop.org/svn/ntop/trunk/attic/PF_RING/userland/snort/pfring-daq-module-zc/README.1st >>>>>>>> >>>>>>>> <https://svn.ntop.org/svn/ntop/trunk/attic/PF_RING/userland/snort/pfring-daq-module-zc/README.1st> >>>>>>>> >>>>>>>> But this does not: >>>>>>>> http://www.ntop.org/pf_ring/accelerating-snort-with-pf_ring-dna/ >>>>>>>> <http://www.ntop.org/pf_ring/accelerating-snort-with-pf_ring-dna/> >>>>>>>> >>>>>>>> On 2 December 2015 at 14:01, James <[email protected] >>>>>>>> <mailto:[email protected]>> wrote: >>>>>>>> Hi all, >>>>>>>> >>>>>>>> I posted a few weeks ago and have since got pf_ring with ZC working. >>>>>>>> I'm now trying to decide how best to configure snort (in IDS mode). My >>>>>>>> server has 4 X 12 core CPU's and two NIC's which are being fed one >>>>>>>> half each of a 10Gb connection. >>>>>>>> >>>>>>>> I have a few key questions: >>>>>>>> - Within the ixgbe zc load_drive.sh script, would the default 16 queue >>>>>>>> option do, or would you choose something different: insmod ./ixgbe.ko >>>>>>>> MQ=1,1,1,1 RSS=16,16,16,16 >>>>>>>> >>>>>>>> - Assuming the choice of 16 above, should I start 16 copies of Snort >>>>>>>> like this (variation on the example from ntop website)? >>>>>>>> snort -q --pid-path /var/run --create-pidfile -D -c >>>>>>>> /etc/snort/snort.conf -l /var/log/snort/eth4_eth5/instance-1 >>>>>>>> --daq-dir=/usr/local/lib/daq --daq pfring_zc --daq-mode passive -i >>>>>>>> zc:eth4@0+zc:eth5@0 --daq-var idsbridge=1 --daq-var bindcpu=0 >>>>>>>> >>>>>>>> The information on http://www.metaflows.com/features/pf_ring >>>>>>>> <http://www.metaflows.com/features/pf_ring> about CPU affinity and >>>>>>>> interrupts has confused me somewhat. >>>>>>>> >>>>>>>> Thanks >>>>>>>> J. >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Ntop-misc mailing list >>>>>>>> [email protected] <mailto:[email protected]> >>>>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>>>>>>> <http://listgateway.unipi.it/mailman/listinfo/ntop-misc> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Ntop-misc mailing list >>>>>>> [email protected] <mailto:[email protected]> >>>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>>>>>> <http://listgateway.unipi.it/mailman/listinfo/ntop-misc> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Ntop-misc mailing list >>>>>>> [email protected] <mailto:[email protected]> >>>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>>>>>> <http://listgateway.unipi.it/mailman/listinfo/ntop-misc> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Ntop-misc mailing list >>>>> [email protected] <mailto:[email protected]> >>>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>>>> <http://listgateway.unipi.it/mailman/listinfo/ntop-misc> >>>>> >>>>> _______________________________________________ >>>>> Ntop-misc mailing list >>>>> [email protected] <mailto:[email protected]> >>>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>>>> <http://listgateway.unipi.it/mailman/listinfo/ntop-misc> >>>> >>>> _______________________________________________ >>>> Ntop-misc mailing list >>>> [email protected] <mailto:[email protected]> >>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>>> <http://listgateway.unipi.it/mailman/listinfo/ntop-misc> >>>> >>>> _______________________________________________ >>>> Ntop-misc mailing list >>>> [email protected] <mailto:[email protected]> >>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>>> <http://listgateway.unipi.it/mailman/listinfo/ntop-misc> >>> >>> _______________________________________________ >>> Ntop-misc mailing list >>> [email protected] <mailto:[email protected]> >>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>> <http://listgateway.unipi.it/mailman/listinfo/ntop-misc> >>> >>> _______________________________________________ >>> Ntop-misc mailing list >>> [email protected] <mailto:[email protected]> >>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>> <http://listgateway.unipi.it/mailman/listinfo/ntop-misc> >> >> _______________________________________________ >> Ntop-misc mailing list >> [email protected] <mailto:[email protected]> >> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >> <http://listgateway.unipi.it/mailman/listinfo/ntop-misc> >> >> _______________________________________________ >> Ntop-misc mailing list >> [email protected] <mailto:[email protected]> >> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >> <http://listgateway.unipi.it/mailman/listinfo/ntop-misc> > > _______________________________________________ > Ntop-misc mailing list > [email protected] <mailto:[email protected]> > http://listgateway.unipi.it/mailman/listinfo/ntop-misc > <http://listgateway.unipi.it/mailman/listinfo/ntop-misc> > > > _______________________________________________ > Ntop-misc mailing list > [email protected] > http://listgateway.unipi.it/mailman/listinfo/ntop-misc
_______________________________________________ Ntop-misc mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop-misc
