Hi Alfredo, For the lists benefit, I have (with your help) solved my problem by increasing $HUGEPAGES to 2048 in the load_driver script. I now have 16 instances of Snort running and alerts are being generated. I am surprised by how few alerts are occurring though and wanted to verify what traffic was being seen. I thought tcpdump would be a good way to do this, so compiled the pf_ring version, but I can't get that to display any traffic on either of my two interfaces. I've tried: "tcpdump -i eth4" (or 5) which functions but shows no traffic "tcpdump -i zc:eth4" "tcpdump -i zc:eth4@0" both of which fail, telling me there is no such device
Sorry for asking so many questions, but any ideas please? Thanks James On 9 December 2015 at 14:40, Alfredo Cardigliano <[email protected]> wrote: > Hi James > for debug purposes, do you see any improvement if you change the number of > ring slots from 32k to 4k for instance? (i.e. ethtool -G ethX rx 4096) > > Alfredo > > On 07 Dec 2015, at 15:15, James <[email protected]> wrote: > > I'm another step forward now, thank you Alfredo. I can get 12 instances to > start (for some reason it's always instances 0-7 and 12-15). Instances 8-11 > give the same "bus error" at startup though. There are no errors in > /var/log/messages > > > > On 4 December 2015 at 11:03, James <[email protected]> wrote: > >> Thanks, that's now set and now 8 start, 4 fail with bus error, 4 more >> start and then back to command line. /var/log/messages no longer mentions >> hugepages, so I guess that's not the problem any longer. The NIC's appear >> to go up and down repeatedly (that might be correct?), come in and out of >> promiscuous mode and then lots of these: >> >> ZC[8537]: error unlink'ing /mnt/huge/pfring_zc_0: Permission denied >> ZC[8549]: error unlink'ing /mnt/huge/pfring_zc_1: Permission denied >> ZC[8568]: error unlink'ing /mnt/huge/pfring_zc_2: Permission denied >> etc >> >> >> On 4 December 2015 at 09:46, Alfredo Cardigliano <[email protected]> >> wrote: >> >>> node means NUMA node (i.e. CPU) >>> >>> Alfredo >>> >>> >>> On 04 Dec 2015, at 10:41, James <[email protected]> wrote: >>> >>> Hopefully my last stupid question - is node the same as processes and >>> queues in this context? So I should do that all the way up to 15? >>> >>> On 4 December 2015 at 09:32, Alfredo Cardigliano <[email protected]> >>> wrote: >>> >>>> >>>> On 04 Dec 2015, at 10:27, James <[email protected]> wrote: >>>> >>>> Thanks Alfredo, even more that you reply so quickly! I respect the >>>> "teach a man to fish.." method of helping, but that's a lot of parameters >>>> and options and I'd be making complete guesses at which ones to change and >>>> to what values. Would it be possible to recommend what you'd change based >>>> on the spec of my system? >>>> >>>> >>>> In essence if you have 4 nodes, you should set the number of huge pages >>>> per node with: >>>> >>>> $ echo 1024 > >>>> /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages >>>> $ echo 1024 > >>>> /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages >>>> $ echo 1024 > >>>> /sys/devices/system/node/node2/hugepages/hugepages-2048kB/nr_hugepages >>>> $ echo 1024 > >>>> /sys/devices/system/node/node3/hugepages/hugepages-2048kB/nr_hugepages >>>> >>>> I've also just noticed that the load_drive script should be changed to >>>> "insmod >>>> ./ixgbe.ko RSS=16,16" because I'm only monitoring two NIC's, is that >>>> correct? >>>> >>>> >>>> Correct >>>> >>>> Alfredo >>>> >>>> >>>> Thank you again >>>> J. >>>> >>>> On 4 December 2015 at 09:04, Alfredo Cardigliano <[email protected]> >>>> wrote: >>>> >>>>> Please note the total amount of pages is divided by the nodes, please >>>>> take a look at >>>>> https://github.com/ntop/PF_RING/blob/dev/doc/README.hugepages >>>>> >>>>> Alfredo >>>>> >>>>> On 04 Dec 2015, at 10:00, James <[email protected]> wrote: >>>>> >>>>> In case it helps anyone else reading this, my startup script needed >>>>> some corrections, ending up with: >>>>> >>>>> for i in `seq 0 1 15`; do >>>>> snort -q -u snort -g snort --pid-path /var/run --create-pidfile -D -c >>>>> /etc/snort/snort.conf -l /logs/snort/eth4_eth5/instance-$i >>>>> --daq-dir=/usr/local/lib/daq --daq pfring_zc --daq-mode passive -i >>>>> zc:eth4@ >>>>> $i,zc:eth5@$i --daq-var clusterid=$i --daq-var bindcpu=$i >>>>> done >>>>> >>>>> i.e. I needed to remove idsbridge=1 (might be a mistake that the IDS >>>>> with multiqueue example does have this set in README.1st I linked >>>>> earlier?) >>>>> and I needed to change my variable to be $i instead of $1. >>>>> >>>>> However when I run this script it only start's 4 "daemon child", then >>>>> gives a "bus error" on the next 4 and then returns to the command line >>>>> with >>>>> no mention of the other 8. /var/log/messages tells me: >>>>> snort[4888]: FATAL ERROR: Can't initialize DAQ pfring_zc (-1) - >>>>> pfring_zc_daq_initialize: Cluster failed: No buffer space available (error >>>>> 105) >>>>> ZC[4897]: error mmap'ing hugepage /mnt/huge/pfring_zc_14: Cannot >>>>> allocate memory >>>>> ZC[4897]: error mmap'ing 128 hugepages of 2048 KB >>>>> >>>>> If it's relevant, when I run the ZC load_driver script (I took the >>>>> MQ=1,1,1,1 >>>>> out as advised, so that just has "insmod ./ixgbe.ko RSS=16,16,16,16") that >>>>> says: >>>>> Warning: 512 hugepages available, 1024 requested >>>>> >>>>> Things I have checked are: >>>>> sudo more /sys/kernel/mm/transparent_hugepage/enabled >>>>> always madvise [never] >>>>> >>>>> sudo more /proc/sys/vm/nr_hugepages >>>>> 1024 >>>>> >>>>> sudo more /proc/meminfo >>>>> MemTotal: 32748700 kB >>>>> MemFree: 27563604 kB >>>>> <snip> >>>>> AnonHugePages: 0 kB >>>>> HugePages_Total: 1024 >>>>> HugePages_Free: 1024 >>>>> HugePages_Rsvd: 0 >>>>> HugePages_Surp: 0 >>>>> Hugepagesize: 2048 kB >>>>> >>>>> Any ideas on how I can fix this please? >>>>> >>>>> Thanks >>>>> J. >>>>> >>>>> On 2 December 2015 at 15:10, Alfredo Cardigliano <[email protected] >>>>> > wrote: >>>>> >>>>>> >>>>>> On 02 Dec 2015, at 16:08, James <[email protected]> wrote: >>>>>> >>>>>> Ah.. so if only 16 queues, I should go back to only 16 copies of >>>>>> snort? >>>>>> >>>>>> >>>>>> Yes, or you can consider using zbalance_ipc for load balance in >>>>>> software to more queues, >>>>>> but I do not what is the performance you can reach with 48 queues, >>>>>> you should run some test. >>>>>> >>>>>> Alfredo >>>>>> >>>>>> >>>>>> On 2 December 2015 at 15:06, Alfredo Cardigliano < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Sorry, forgot to tell you RSS on ixgbe supports up to 16 queues. >>>>>>> >>>>>>> Alfredo >>>>>>> >>>>>>> On 02 Dec 2015, at 16:06, Alfredo Cardigliano <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>> It looks fine, you can omit MQ. >>>>>>> >>>>>>> Alfredo >>>>>>> >>>>>>> On 02 Dec 2015, at 16:04, James <[email protected]> wrote: >>>>>>> >>>>>>> Many thanks for the help Alfredo. So I'll crank things up to use all >>>>>>> CPU's and that gives me (I've converted it to a for loop): >>>>>>> >>>>>>> for i in `seq 0 1 48`; do >>>>>>> snort -q --pid-path /var/run --create-pidfile -D -c >>>>>>> /etc/snort/snort.conf -l /logs/snort/eth4_eth5/instance-$1 >>>>>>> --daq-dir=/usr/local/lib/daq --daq pfring_zc --daq-mode passive -i >>>>>>> zc:eth4@ >>>>>>> $1,zc:eth5@$1 --daq-var clusterid=$1 --daq-var idsbridge=1 >>>>>>> --daq-var bindcpu=$1 >>>>>>> done >>>>>>> >>>>>>> Is this the correct load_driver.sh setting to match? I'm not sure >>>>>>> about the MQ values? >>>>>>> insmod ./ixgbe.ko MQ=1,1,1,1 RSS=48,48,48,48 >>>>>>> >>>>>>> On 2 December 2015 at 14:15, Alfredo Cardigliano < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Please use README.1st as reference. >>>>>>>> What you need to know: >>>>>>>> 1. Use --daq-var clusterid=K where K is a unique number per snort >>>>>>>> instance, used for resource allocation >>>>>>>> 2. Use --daq-var bindcpu=K where K is the core id for affinity, >>>>>>>> please ignore interrupts affinity with ZC >>>>>>>> 3. Use “,” in -i in please of “+” for interfaces aggregation, “+” >>>>>>>> is used for IPS/IDS-bridge mode >>>>>>>> 4. We usually recommend using only the CPU where the NIC is >>>>>>>> connected, however since snort is (likely) the bottleneck, feel free >>>>>>>> to use >>>>>>>> all the cores available, setting RSS=N,N where N is the number of >>>>>>>> cores and >>>>>>>> the number of snort instances. >>>>>>>> >>>>>>>> Alfredo >>>>>>>> >>>>>>>> On 02 Dec 2015, at 15:08, James <[email protected]> wrote: >>>>>>>> >>>>>>>> Follow-up question - should I use the cluster-id parameter? >>>>>>>> >>>>>>>> This uses it: >>>>>>>> >>>>>>>> https://svn.ntop.org/svn/ntop/trunk/attic/PF_RING/userland/snort/pfring-daq-module-zc/README.1st >>>>>>>> >>>>>>>> But this does not: >>>>>>>> http://www.ntop.org/pf_ring/accelerating-snort-with-pf_ring-dna/ >>>>>>>> >>>>>>>> On 2 December 2015 at 14:01, James <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi all, >>>>>>>>> >>>>>>>>> I posted a few weeks ago and have since got pf_ring with ZC >>>>>>>>> working. I'm now trying to decide how best to configure snort (in IDS >>>>>>>>> mode). My server has 4 X 12 core CPU's and two NIC's which are being >>>>>>>>> fed >>>>>>>>> one half each of a 10Gb connection. >>>>>>>>> >>>>>>>>> I have a few key questions: >>>>>>>>> - Within the ixgbe zc load_drive.sh script, would the default 16 >>>>>>>>> queue option do, or would you choose something different: insmod >>>>>>>>> ./ixgbe.ko >>>>>>>>> MQ=1,1,1,1 RSS=16,16,16,16 >>>>>>>>> >>>>>>>>> - Assuming the choice of 16 above, should I start 16 copies of >>>>>>>>> Snort like this (variation on the example from ntop website)? >>>>>>>>> snort -q --pid-path /var/run --create-pidfile -D -c >>>>>>>>> /etc/snort/snort.conf -l /var/log/snort/eth4_eth5/instance-1 >>>>>>>>> --daq-dir=/usr/local/lib/daq --daq pfring_zc --daq-mode passive -i >>>>>>>>> zc:eth4@0 >>>>>>>>> +zc:eth5@0 --daq-var idsbridge=1 --daq-var bindcpu=0 >>>>>>>>> >>>>>>>>> The information on http://www.metaflows.com/features/pf_ring >>>>>>>>> about CPU affinity and interrupts has confused me somewhat. >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> J. >>>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Ntop-misc mailing list >>>>>>>> [email protected] >>>>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Ntop-misc mailing list >>>>>>>> [email protected] >>>>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Ntop-misc mailing list >>>>>>> [email protected] >>>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Ntop-misc mailing list >>>>>>> [email protected] >>>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Ntop-misc mailing list >>>>>> [email protected] >>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Ntop-misc mailing list >>>>>> [email protected] >>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>>>>> >>>>> >>>>> _______________________________________________ >>>>> Ntop-misc mailing list >>>>> [email protected] >>>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Ntop-misc mailing list >>>>> [email protected] >>>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>>>> >>>> >>>> _______________________________________________ >>>> Ntop-misc mailing list >>>> [email protected] >>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>>> >>>> >>>> >>>> _______________________________________________ >>>> Ntop-misc mailing list >>>> [email protected] >>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>>> >>> >>> _______________________________________________ >>> Ntop-misc mailing list >>> [email protected] >>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>> >>> >>> >>> _______________________________________________ >>> Ntop-misc mailing list >>> [email protected] >>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >>> >> >> > _______________________________________________ > Ntop-misc mailing list > [email protected] > http://listgateway.unipi.it/mailman/listinfo/ntop-misc > > > > _______________________________________________ > Ntop-misc mailing list > [email protected] > http://listgateway.unipi.it/mailman/listinfo/ntop-misc >
_______________________________________________ Ntop-misc mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop-misc
