Chris you can dynamically allocate hugepages with: echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
or you can change the default size and allocate them at boot time with kernel parameters: default_hugepagesz=1G hugepagesz=1G hugepages=4 In order to make the hugepages available for use, before running the cluster, you have to mount the hugetlbfs filesystem somewhere: mount -t hugetlbfs none /mnt/hugepages After that, you can run the pfdnacluster_master with -u. Please let us know. Regards Alfredo On Oct 23, 2012, at 7:09 PM, Chris Wakelin <[email protected]> wrote: > I'd not heard of hugepages before. It looks like Ubuntu 12.04 has 2MB > pages defined but none allocated. It also supports "transparent huge > pages". Can these be used by DNA clusters? Can you give some suggested > settings I could try? > > Best Wishes, > Chris > > On 21/10/12 18:51, Alfredo Cardigliano wrote: >> Chris >> besides the low-level settings, now you can also use the hugepages support. >> Be aware it is a pretty new feature under testing (see pfdnacluster_master >> -u). >> Please note that you have to setup the hugepages support in your system >> before using it. >> If you decide to have a look at it, please let us know if it is working fine >> for you. >> >> Best Regards >> Alfredo >> >> On Oct 18, 2012, at 10:03 PM, Chris Wakelin <[email protected]> >> wrote: >> >>> Thanks. Now running with 15 queues + 1 duplicate of everything, with >>> additional buffers disabled. >>> >>> It still won't manage 2x15 + 1 though. I'll need to experiment with the >>> low-level settings. >>> >>> Best Wishes, >>> Chris >>> >>> On 18/10/12 19:41, Alfredo Cardigliano wrote: >>>> Chris >>>> the segfault has been fixed, please update from SVN. >>>> It was only with the multi-process cluster. >>>> >>>> Thank you >>>> Alfredo >>>> >>>> On Oct 18, 2012, at 8:20 PM, Alfredo Cardigliano <[email protected]> >>>> wrote: >>>> >>>>> >>>>> On Oct 18, 2012, at 8:03 PM, Chris Wakelin <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi Alfredo, >>>>>> >>>>>> Thanks for the extra options. However, I updated to SVN 5758 and now >>>>>> pfcount (and pfcount_aggregator) segfaults talking to a cluster (started >>>>>> with "pfdnacluster_master -i dna0 -c 1 -n 12") >>>>>> >>>>>>> Core was generated by `/opt/RDGpfring/bin/pfcount -i dnacl:1@0'. >>>>>>> Program terminated with signal 11, Segmentation fault. >>>>>>> #0 0x000000000040e212 in pfring_dna_cluster_open () >>>>>>> (gdb) bt full >>>>>>> #0 0x000000000040e212 in pfring_dna_cluster_open () >>>>>>> No symbol table info available. >>>>>>> #1 0x00000000004087d3 in pfring_open () >>>>>>> No symbol table info available. >>>>>>> #2 0x0000000000406a7b in main () >>>>>>> No symbol table info available. >>>>>> >>>>>> pfcount on plain DNA is fine. >>>>> >>>>> Ok, I'll try to reproduce this. >>>>> >>>>>> >>>>>> A couple of other questions :-) :- >>>>>> >>>>>> 1) I see there's also a DNA_CLUSTER_DIRECT_FORWARDING flag. I'm guessing >>>>>> that's for disabling the wait-for-apps-to-retrieve-packets feature? Does >>>>>> it save memory? >>>>> >>>>> No, you can use this flag to send a packet to an interface directly from >>>>> the distribution function. >>>>> Actually it requires more memory (not so much). >>>>> >>>>>> >>>>>> 2) What are slave rx and tx slots? If I'm only interested in receiving >>>>>> packets, can tx slots be 0? >>>>> >>>>> Number of slots of the the master <-> consumer queues. >>>>> The minimum is 512 slots, however if you don't need tx you can set the >>>>> recv_only_mode (no tx memory allocation this way). >>>>> >>>>>> >>>>>> 3) What are the additional buffers used for? Does disabling them break >>>>>> anything? >>>>> >>>>> They are used for extra buffer allocation with pfring_alloc_pkt_buff() / >>>>> pfring_release_pkt_buff() >>>>> (for instance if you want to put a packet aside for future access and >>>>> recv the next one with pfring_recv_pkt_buff()). >>>>> Usually you don't need them. >>>>> >>>>> Regards >>>>> Alfredo >>>>> >>>>>> >>>>>> Best Wishes, >>>>>> Chris >>>>>> >>>>>> On 17/10/12 23:38, Alfredo Cardigliano wrote: >>>>>>> Chris >>>>>>> with latest code from SVN it is now possible to: >>>>>>> - Disable the allocation of additional per-consumer buffers (those >>>>>>> available with alloc()/release()) via the dna_cluster_create() flags: >>>>>>> dna_cluster_create(cluster_id, num_apps, >>>>>>> DNA_CLUSTER_NO_ADDITIONAL_BUFFERS); >>>>>>> >>>>>>> - Configure the number of per-slave rx/tx queue slots and additional >>>>>>> buffers: >>>>>>> dna_cluster_low_level_settings(dna_cluster_handle, 8192 /* rx queue >>>>>>> slots */, 1024 /* tx queue slots */, 0 /* additional buffers */); >>>>>>> (call this just after dna_cluster_create()) >>>>>>> >>>>>>> Regards >>>>>>> Alfredo >>>>>>> >>>>>>> On Oct 17, 2012, at 8:10 PM, Alfredo Cardigliano <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> >>>>>>>> On Oct 17, 2012, at 7:27 PM, Chris Wakelin <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> On 17/10/12 17:39, Alfredo Cardigliano wrote: >>>>>>>>>> Chris >>>>>>>>>> please see inline >>>>>>>>>> >>>>>>>>>> On Oct 17, 2012, at 6:00 PM, Chris Wakelin >>>>>>>>>> <[email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> I still can't get more than 12 cores used with Suricata on my Ubuntu >>>>>>>>>>> 12.04 machine with ixgbe. Even with DNA + RSS and Suricata using >>>>>>>>>>> dna0@0 >>>>>>>>>>> ... dna0@15, it fails for pfring_open on dna0@12 to dna0@15 (though >>>>>>>>>>> pfcount_aggregator manages the 16 queues in that case). >>>>>>>>>> >>>>>>>>>> You mean standard DNA (no DNA cluster, etc), right? >>>>>>>>>> This is definitely strange as DNA memory is allocated when loading >>>>>>>>>> the driver. >>>>>>>>> >>>>>>>>> Yes I meant standard DNA. >>>>>>>>> >>>>>>>>> Hmm. Strangely it's working now! Last night it didn't but I can't see >>>>>>>>> why. I tried again this morning but thought it failed, when now it >>>>>>>>> seems >>>>>>>>> it didn't (silly me). I was probably mistaken about the discrepancy >>>>>>>>> with >>>>>>>>> pfcount_aggregator. >>>>>>>>> >>>>>>>>>> >>>>>>>>>>> How is memory allocated in DNA? Are there kernel options I'm >>>>>>>>>>> missing? >>>>>>>>>> >>>>>>>>>> No, there is no configuration for that. >>>>>>>>>> >>>>>>>>>>> With DNA clusters, I can't get pfdnacluster_master to manage more >>>>>>>>>>> than >>>>>>>>>>> 16 queues either. I would have expected my custom one with >>>>>>>>>>> duplication >>>>>>>>>>> should actually only use as much memory as it does without >>>>>>>>>>> duplication >>>>>>>>>>> as the duplicates are of course the same packets and therefore the >>>>>>>>>>> same >>>>>>>>>>> memory. >>>>>>>>>> >>>>>>>>>> Even if you are using duplication, memory with DNA clusters is >>>>>>>>>> allocated when opening the socket. >>>>>>>>>> Actually on my test system with 4Gb of RAM I can run up to two >>>>>>>>>> cluster with 32 queues each. >>>>>>>>>> Anyway memory management in libzero is something we are working on >>>>>>>>>> (there is space for improvements). >>>>>>>>> >>>>>>>>> Is that with all the sockets open? I can certainly start >>>>>>>>> pfdnacluster_master with that many queues, but the applications fail. >>>>>>>> >>>>>>>> Yes, up and running. I forgot to tell you I'm using the default >>>>>>>> num_rx_slots/num_tx_slots (I don't know if you are using higher >>>>>>>> values). >>>>>>>> >>>>>>>>> How much memory is used per socket? Strangely I didn't have problems >>>>>>>>> with testing e1000e DNA +libzero on another 64-bit system with less >>>>>>>>> memory (16GB instead of 32) but running Ubuntu 10.04 instead of 12.04. >>>>>>>>> >>>>>>>>> Is the memory used what is shown in ifconfig? :- >>>>>>>> >>>>>>>> No >>>>>>>> >>>>>>>>> >>>>>>>>> dna0 Link encap:Ethernet HWaddr 00:1b:21:cd:a2:74 >>>>>>>>> inet6 addr: fe80::21b:21ff:fecd:a274/64 Scope:Link >>>>>>>>> UP BROADCAST RUNNING PROMISC MULTICAST MTU:1522 Metric:1 >>>>>>>>> RX packets:195271292 errors:0 dropped:0 overruns:0 frame:0 >>>>>>>>> TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 >>>>>>>>> collisions:0 txqueuelen:1000 >>>>>>>>> RX bytes:157968626881 (157.9 GB) TX bytes:0 (0.0 B) >>>>>>>>> Memory:d9280000-d9300000 >>>>>>>>> >>>>>>>>> Does the setting of "ethtool -g" make a difference (presumably the >>>>>>>>> same >>>>>>>>> as num_rx_slots= in the module parameters)? >>>>>>>> >>>>>>>> No, ethtool -g is not supported. >>>>>>>> >>>>>>>>> >>>>>>>>> Sorry for the inquisition :-) but it would be nice to understand >>>>>>>>> what's >>>>>>>>> possible. >>>>>>>> >>>>>>>> np >>>>>>>> >>>>>>>> Alfredo >>>>>>>> >>>>>>>>> >>>>>>>>> Best Wishes, >>>>>>>>> Chris > -- > --+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+- > Christopher Wakelin, [email protected] > IT Services Centre, The University of Reading, Tel: +44 (0)118 378 2908 > Whiteknights, Reading, RG6 6AF, UK Fax: +44 (0)118 975 3094 > _______________________________________________ > Ntop-misc mailing list > [email protected] > http://listgateway.unipi.it/mailman/listinfo/ntop-misc _______________________________________________ Ntop-misc mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop-misc
