I still can't get more than 12 cores used with Suricata on my Ubuntu 12.04 machine with ixgbe. Even with DNA + RSS and Suricata using dna0@0 ... dna0@15, it fails for pfring_open on dna0@12 to dna0@15 (though pfcount_aggregator manages the 16 queues in that case).
How is memory allocated in DNA? Are there kernel options I'm missing? With DNA clusters, I can't get pfdnacluster_master to manage more than 16 queues either. I would have expected my custom one with duplication should actually only use as much memory as it does without duplication as the duplicates are of course the same packets and therefore the same memory. Best Wishes, Chris On 08/10/12 13:21, Chris Wakelin wrote: > OK loaded with just "insmod ixgbe.ko RSS=1,1 mtu=1522", and it works for > "-n 12". If I use my custom pfdnacluster_master that duplicates traffic > to another 12 queues, then it fails again (I guess I should try the > original pfdnacluster_master with "-n 24"). > > It would be useful to know what the memory limitations are! > > Anyway, I'm running Suricata on 12 cores at the moment, which is the > important thing. > > Best Wishes, > Chris > > On 08/10/12 13:00, Alfredo Cardigliano wrote: >> >> On Oct 8, 2012, at 1:56 PM, Chris Wakelin <[email protected]> wrote: >> >>> Actually I'm wondering about the FdirPballoc; I'm not actually using >>> hardware filters at the moment (though may want to in the future), so I >>> can probably have FdirMode=0? >> >> Yes >> >> Alfredo >> >>> >>> Best Wishes, >>> Chris >>> >>> On 08/10/12 12:51, Alfredo Cardigliano wrote: >>>> >>>> On Oct 8, 2012, at 1:46 PM, Chris Wakelin <[email protected]> >>>> wrote: >>>> >>>>> I've got 32GB, but it's a 3.2.0 kernel (Ubuntu 12.04 64-bit). I tried >>>>> >>>>> insmod ixgbe.ko RSS=1,1 FdirPballoc=3,3 mtu=1522 num_rx_slots=2048 >>>>> num_tx_slots=0 >>>> >>>> Ok, I will check the same kernel version/driver options. >>>> >>>>> >>>>> but it didn't seem to make any difference. >>>>> >>>>> pf_ring is loaded with "transparent_mode=2 min_num_slots=4096" >>>> >>>> These settings are ignored with DNA. >>>> >>>> Alfredo >>>> >>>>> >>>>> Best Wishes, >>>>> Chris >>>>> >>>>> On 08/10/12 12:35, Alfredo Cardigliano wrote: >>>>>> Christ >>>>>> How much memory do you have on this system? >>>>>> On my system (kernel 2.6.36, 4Gb of RAM, ixgbe) it is working fine. >>>>>> Please try decreasing the num_rx_slots (actually memory allocation >>>>>> happens at pfdnacluster_master boot, but it's worth giving it a try >>>>>> because it's possible you are running out of memory). >>>>>> >>>>>> Regards >>>>>> Alfredo >>>>>> >>>>>> On Oct 8, 2012, at 1:17 PM, Chris Wakelin <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Any ideas on this? Is this related to the 32768 num_rx_slots on ixgbe? >>>>>>> >>>>>>> I'd like to get somewhere with this as with just 8 cores, I'm dropping >>>>>>> packets. >>>>>>> >>>>>>> Best Wishes, >>>>>>> Chris >>>>>>> >>>>>>> On 01/10/12 17:50, Chris Wakelin wrote: >>>>>>>> Hi, >>>>>>>> >>>>>>>> I've got a strange problem when trying to increase the number of cores >>>>>>>> used by my Suricata+Bro IDS(+ARGUS) setup. When I try to get Suricata >>>>>>>> to >>>>>>>> use more than 8 devices, it fails on my Ubuntu 12.04 (kernel 3.2.0) >>>>>>>> boxes. However on my Ubuntu 10.04 (kernel 2.6.32) box it seems OK. >>>>>>>> >>>>>>>> Strangely, there's no problem running Suricata, using libpfring, on >>>>>>>> dnacluster:1@0 .. 1@7 and Bro IDS, using lipcap, on dnacluster:1@8 .. >>>>>>>> 1@15 (with my custom pfdnacluster_master that duplicates traffic). >>>>>>>> >>>>>>>> I can reproduce the problem with pfcount_aggregator (with "#define >>>>>>>> MAX_NUM_DEVS 16" rather than 8):- >>>>>>>> >>>>>>>>> pfdnacluster_master -i dna0 -c 1 -n 12 >>>>>>>> >>>>>>>>> pfcount_aggregator_cdw -i >>>>>>>>> dnacluster:1@0+dnacluster:1@1+dnacluster:1@2+dnacluster:1@3+dnacluster:1@4+dnacluster:1@5+dnacluster:1@6+dnacluster:1@7+dnacluster:1@8+dnacluster:1@9+dnacluster:1@10+dnacluster:1@11 >>>>>>>>> -l 1522 >>>>>>>>> Using PF_RING v.5.4.5 >>>>>>>>> Impossible to know the device address >>>>>>>>> # Device RX channels: 1 >>>>>>>>> pfring_set_direction returned [rc=-7][direction=0] >>>>>>>>> Impossible to know the device address >>>>>>>>> # Device RX channels: 1 >>>>>>>>> pfring_set_direction returned [rc=-7][direction=0] >>>>>>>>> Impossible to know the device address >>>>>>>>> # Device RX channels: 1 >>>>>>>>> pfring_set_direction returned [rc=-7][direction=0] >>>>>>>>> Impossible to know the device address >>>>>>>>> # Device RX channels: 1 >>>>>>>>> pfring_set_direction returned [rc=-7][direction=0] >>>>>>>>> Impossible to know the device address >>>>>>>>> # Device RX channels: 1 >>>>>>>>> pfring_set_direction returned [rc=-7][direction=0] >>>>>>>>> Impossible to know the device address >>>>>>>>> # Device RX channels: 1 >>>>>>>>> pfring_set_direction returned [rc=-7][direction=0] >>>>>>>>> Impossible to know the device address >>>>>>>>> # Device RX channels: 1 >>>>>>>>> pfring_set_direction returned [rc=-7][direction=0] >>>>>>>>> Impossible to know the device address >>>>>>>>> # Device RX channels: 1 >>>>>>>>> pfring_set_direction returned [rc=-7][direction=0] >>>>>>>>> pfring_open error [Cannot allocate memory] (pf_ring not loaded or >>>>>>>>> perhaps you use quick mode and have already a socket bound to >>>>>>>>> dnacluster:1@8 ?) >>>>>>>> >>>>>>>> >>>>>>>> The 12.04 boxes have ixgbe devices loaded with "insmod ixgbe.ko RSS=1,1 >>>>>>>> FdirPballoc=3,3 mtu=1522". >>>>>>>> >>>>>>>> The 10.04 box has e1000e (DNA demo license) loaded with just "insmod >>>>>>>> e1000e". >>>>>>>> >>>>>>>> Is this expected? >>>>>>>> >>>>>>>> Best Wishes, >>>>>>>> Chris -- --+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+- Christopher Wakelin, [email protected] IT Services Centre, The University of Reading, Tel: +44 (0)118 378 2908 Whiteknights, Reading, RG6 6AF, UK Fax: +44 (0)118 975 3094 _______________________________________________ Ntop-misc mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop-misc
