Chris please see inline On Oct 17, 2012, at 6:00 PM, Chris Wakelin <[email protected]> wrote:
> I still can't get more than 12 cores used with Suricata on my Ubuntu > 12.04 machine with ixgbe. Even with DNA + RSS and Suricata using dna0@0 > ... dna0@15, it fails for pfring_open on dna0@12 to dna0@15 (though > pfcount_aggregator manages the 16 queues in that case). You mean standard DNA (no DNA cluster, etc), right? This is definitely strange as DNA memory is allocated when loading the driver. > How is memory allocated in DNA? Are there kernel options I'm missing? No, there is no configuration for that. > With DNA clusters, I can't get pfdnacluster_master to manage more than > 16 queues either. I would have expected my custom one with duplication > should actually only use as much memory as it does without duplication > as the duplicates are of course the same packets and therefore the same > memory. Even if you are using duplication, memory with DNA clusters is allocated when opening the socket. Actually on my test system with 4Gb of RAM I can run up to two cluster with 32 queues each. Anyway memory management in libzero is something we are working on (there is space for improvements). Regards Alfredo > > Best Wishes, > Chris > > On 08/10/12 13:21, Chris Wakelin wrote: >> OK loaded with just "insmod ixgbe.ko RSS=1,1 mtu=1522", and it works for >> "-n 12". If I use my custom pfdnacluster_master that duplicates traffic >> to another 12 queues, then it fails again (I guess I should try the >> original pfdnacluster_master with "-n 24"). >> >> It would be useful to know what the memory limitations are! >> >> Anyway, I'm running Suricata on 12 cores at the moment, which is the >> important thing. >> >> Best Wishes, >> Chris >> >> On 08/10/12 13:00, Alfredo Cardigliano wrote: >>> >>> On Oct 8, 2012, at 1:56 PM, Chris Wakelin <[email protected]> wrote: >>> >>>> Actually I'm wondering about the FdirPballoc; I'm not actually using >>>> hardware filters at the moment (though may want to in the future), so I >>>> can probably have FdirMode=0? >>> >>> Yes >>> >>> Alfredo >>> >>>> >>>> Best Wishes, >>>> Chris >>>> >>>> On 08/10/12 12:51, Alfredo Cardigliano wrote: >>>>> >>>>> On Oct 8, 2012, at 1:46 PM, Chris Wakelin <[email protected]> >>>>> wrote: >>>>> >>>>>> I've got 32GB, but it's a 3.2.0 kernel (Ubuntu 12.04 64-bit). I tried >>>>>> >>>>>> insmod ixgbe.ko RSS=1,1 FdirPballoc=3,3 mtu=1522 num_rx_slots=2048 >>>>>> num_tx_slots=0 >>>>> >>>>> Ok, I will check the same kernel version/driver options. >>>>> >>>>>> >>>>>> but it didn't seem to make any difference. >>>>>> >>>>>> pf_ring is loaded with "transparent_mode=2 min_num_slots=4096" >>>>> >>>>> These settings are ignored with DNA. >>>>> >>>>> Alfredo >>>>> >>>>>> >>>>>> Best Wishes, >>>>>> Chris >>>>>> >>>>>> On 08/10/12 12:35, Alfredo Cardigliano wrote: >>>>>>> Christ >>>>>>> How much memory do you have on this system? >>>>>>> On my system (kernel 2.6.36, 4Gb of RAM, ixgbe) it is working fine. >>>>>>> Please try decreasing the num_rx_slots (actually memory allocation >>>>>>> happens at pfdnacluster_master boot, but it's worth giving it a try >>>>>>> because it's possible you are running out of memory). >>>>>>> >>>>>>> Regards >>>>>>> Alfredo >>>>>>> >>>>>>> On Oct 8, 2012, at 1:17 PM, Chris Wakelin <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Any ideas on this? Is this related to the 32768 num_rx_slots on ixgbe? >>>>>>>> >>>>>>>> I'd like to get somewhere with this as with just 8 cores, I'm dropping >>>>>>>> packets. >>>>>>>> >>>>>>>> Best Wishes, >>>>>>>> Chris >>>>>>>> >>>>>>>> On 01/10/12 17:50, Chris Wakelin wrote: >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> I've got a strange problem when trying to increase the number of cores >>>>>>>>> used by my Suricata+Bro IDS(+ARGUS) setup. When I try to get Suricata >>>>>>>>> to >>>>>>>>> use more than 8 devices, it fails on my Ubuntu 12.04 (kernel 3.2.0) >>>>>>>>> boxes. However on my Ubuntu 10.04 (kernel 2.6.32) box it seems OK. >>>>>>>>> >>>>>>>>> Strangely, there's no problem running Suricata, using libpfring, on >>>>>>>>> dnacluster:1@0 .. 1@7 and Bro IDS, using lipcap, on dnacluster:1@8 .. >>>>>>>>> 1@15 (with my custom pfdnacluster_master that duplicates traffic). >>>>>>>>> >>>>>>>>> I can reproduce the problem with pfcount_aggregator (with "#define >>>>>>>>> MAX_NUM_DEVS 16" rather than 8):- >>>>>>>>> >>>>>>>>>> pfdnacluster_master -i dna0 -c 1 -n 12 >>>>>>>>> >>>>>>>>>> pfcount_aggregator_cdw -i >>>>>>>>>> dnacluster:1@0+dnacluster:1@1+dnacluster:1@2+dnacluster:1@3+dnacluster:1@4+dnacluster:1@5+dnacluster:1@6+dnacluster:1@7+dnacluster:1@8+dnacluster:1@9+dnacluster:1@10+dnacluster:1@11 >>>>>>>>>> -l 1522 >>>>>>>>>> Using PF_RING v.5.4.5 >>>>>>>>>> Impossible to know the device address >>>>>>>>>> # Device RX channels: 1 >>>>>>>>>> pfring_set_direction returned [rc=-7][direction=0] >>>>>>>>>> Impossible to know the device address >>>>>>>>>> # Device RX channels: 1 >>>>>>>>>> pfring_set_direction returned [rc=-7][direction=0] >>>>>>>>>> Impossible to know the device address >>>>>>>>>> # Device RX channels: 1 >>>>>>>>>> pfring_set_direction returned [rc=-7][direction=0] >>>>>>>>>> Impossible to know the device address >>>>>>>>>> # Device RX channels: 1 >>>>>>>>>> pfring_set_direction returned [rc=-7][direction=0] >>>>>>>>>> Impossible to know the device address >>>>>>>>>> # Device RX channels: 1 >>>>>>>>>> pfring_set_direction returned [rc=-7][direction=0] >>>>>>>>>> Impossible to know the device address >>>>>>>>>> # Device RX channels: 1 >>>>>>>>>> pfring_set_direction returned [rc=-7][direction=0] >>>>>>>>>> Impossible to know the device address >>>>>>>>>> # Device RX channels: 1 >>>>>>>>>> pfring_set_direction returned [rc=-7][direction=0] >>>>>>>>>> Impossible to know the device address >>>>>>>>>> # Device RX channels: 1 >>>>>>>>>> pfring_set_direction returned [rc=-7][direction=0] >>>>>>>>>> pfring_open error [Cannot allocate memory] (pf_ring not loaded or >>>>>>>>>> perhaps you use quick mode and have already a socket bound to >>>>>>>>>> dnacluster:1@8 ?) >>>>>>>>> >>>>>>>>> >>>>>>>>> The 12.04 boxes have ixgbe devices loaded with "insmod ixgbe.ko >>>>>>>>> RSS=1,1 >>>>>>>>> FdirPballoc=3,3 mtu=1522". >>>>>>>>> >>>>>>>>> The 10.04 box has e1000e (DNA demo license) loaded with just "insmod >>>>>>>>> e1000e". >>>>>>>>> >>>>>>>>> Is this expected? >>>>>>>>> >>>>>>>>> Best Wishes, >>>>>>>>> Chris > > -- > --+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+- > Christopher Wakelin, [email protected] > IT Services Centre, The University of Reading, Tel: +44 (0)118 378 2908 > Whiteknights, Reading, RG6 6AF, UK Fax: +44 (0)118 975 3094 > _______________________________________________ > Ntop-misc mailing list > [email protected] > http://listgateway.unipi.it/mailman/listinfo/ntop-misc _______________________________________________ Ntop-misc mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop-misc
