-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Well, that shines some light:
cat /proc/meminfo | grep Huge AnonHugePages: 12288 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB I followed the instructions here and set up 512 hugepages: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/5/html/Tuning_and_Optimizing_Red_Hat_Enterprise_Linux_for_Oracle_9i_and_10g_Databases/sect-Oracle_9i_and_10g_Tuning_Guide-Large_Memory_Optimization_Big_Pages_and_Huge_Pages-Configuring_Huge_Pages_in_Red_Hat_Enterprise_Linux_4_or_5.html Now I can run ./zbalance_ipc -i zc:enp4s0 -m 1 -n 12 -c 44 Without specifying a core, I still get the /sys message: libnuma: Warning: /sys not mounted or invalid. Assuming one node: No such file or directory Should I specify a core? I have 12 snorts running like so now: snort -D -i zc:44@0 --daq-dir=/opt/pf/lib/daq --daq pfring_zc \ --daq-var clusterid=44 --daq-var bindcpu=6 -c /etc/snort.conf -l /var/log/snort1 -R 1 snort is reporting low packet loss: Mar 6 09:00:40 sensor snort[24133]: Analyzed: 86462955 (100.000%) Mar 6 09:00:40 sensor snort[24133]: Dropped: 89698 ( 0.104%) but zbalance_ipc is reporting some: Absolute Stats: Recv 1'107'627'115 pkts (0 drops) - Forwarded 1'101'327'836 pkts (6'291'365 drops) Actual Stats: Recv 444'917.60 pps (0.00 drops) - Forwarded 428'529.86 pps (8'799.78 drops) Vanilla pf_ring/snort is reporting high losses on my production sensor: Mar 6 08:21:05 prodsensor snort[13436]: Analyzed: 66520753 (100.000%) Mar 6 08:21:05 prodsensor snort[13436]: Dropped: 26129687 ( 28.202%) So it seems there's an improvement with zc. Do these stats show you anything? Jim On 03/06/2015 04:05 AM, Alfredo Cardigliano wrote: > Hi Jim > can I see your /proc/meminfo | grep Huge ? > > Alfredo > >> On 04 Mar 2015, at 22:06, Jim Hranicky <[email protected] >> <mailto:[email protected]>> wrote: >> >> Signed PGP part >> If I bind it to a core (which seems odd?), the /sys error goes away, >> but I still get the hugepages error: >> >> % ./zbalance_ipc -i zc:enp4s0 -m 1 -n 12 -c 44 -g 1 >> >> From strace: >> >> socket(0x1b /* PF_??? */, SOCK_RAW, 768) = 3 >> setsockopt(3, SOL_IP, 0x87 /* IP_??? */, ",\0\0\0\0\0\0\0", 8) = 0 >> setsockopt(3, SOL_IP, 0x6e /* IP_??? */, "pfring-zc-cluster-44", 20) = 0 >> open("/proc/meminfo", O_RDONLY) = 4 >> fstat(4, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 >> mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, >> -1, 0) = 0x7f05780b1000 >> read(4, "MemTotal: 16251784 kB\nMemF"..., 1024) = 1024 >> read(4, "Total: 0\nHugePages_Free: "..., 1024) = 202 >> read(4, "", 1024) = 0 >> close(4) = 0 >> munmap(0x7f05780b1000, 4096) = 0 >> write(2, " *** error retrieving hugepages "..., 41) = 41 >> close(3) = 0 >> write(2, "pfring_zc_create_cluster error ["..., 99) = 99 >> exit_group(-1) = ? >> >> Anyone have any ideas? I'm trying to set up ZC so 12 snorts >> can read from the ZC interface. Maybe I'm not doing it right? >> >> http://i3.kym-cdn.com/photos/images/facebook/000/234/739/fa5.jpg >> >> Jim >> >> On 02/27/2015 01:37 PM, Jim Hranicky wrote: >> > strace on zbalance_ipc shows this: >> > >> > 10036 open("/sys/devices/system/node/node0/cpumap", O_RDONLY) = 3 >> > 10036 fstat(3, {st_mode=S_IFREG|0444, st_size=4096, ...}) = 0 >> > 10036 mmap(NULL, 4096, PROT_READ|PROT_WRITE, >> MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f0ca6e4d000 >> > 10036 read(3, "00000000,00000fff\n", 4096) = 18 >> > 10036 close(3) = 0 >> > 10036 munmap(0x7f0ca6e4d000, 4096) = 0 >> > 10036 open("/sys/devices/system/node/node1/cpumap", O_RDONLY) = -1 >> ENOENT (No such file or directory) >> > 10036 write(2, "libnuma: Warning: ", 18) = 18 >> > 10036 write(2, "/sys not mounted or invalid. Ass"..., 73) = 73 >> > 10036 write(2, "\n", 1) = 1 >> > >> > However, my nodes are 0 and 2: >> > >> > % numactl --show >> > policy: default >> > preferred node: current >> > physcpubind: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 >> 21 22 23 >> > cpubind: 0 2 >> > nodebind: 0 2 >> > membind: 0 2 >> > >> > % numactl --hardware >> > available: 2 nodes (0,2) >> > node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 >> > node 0 size: 8180 MB >> > node 0 free: 7388 MB >> > node 2 cpus: 12 13 14 15 16 17 18 19 20 21 22 23 >> > node 2 size: 8192 MB >> > node 2 free: 7198 MB >> > node distances: >> > node 0 2 >> > 0: 10 16 >> > 2: 16 10 >> > >> > Is there an easy fix for this? >> > >> > Jim >> > >> > On 02/27/2015 03:51 AM, Alfredo Cardigliano wrote: >> >> Hi Jim >> >> I see several strange problems on this machine, >> >> you should try to solve the libnuma issue (it’s the first time I >> see this wargning) >> >> and the hugepages issue (you should be able to mount hugepages as we do >> >> in the load script, unmounting other hugetlbfs before, if not >> needed, might do the trick), >> >> before running our drivers. >> > >> >> Alfredo >> > >> >>> On 26 Feb 2015, at 23:04, Jim Hranicky <[email protected] >> <mailto:[email protected]>> wrote: >> >>> >> >>> No idea if this has anything to do with it: >> >>> >> >>> % ./zbalance_ipc -i zc:enp4s0 -m 1 -n 12 -c 44 >> >>> libnuma: Warning: /sys not mounted or invalid. Assuming one node: >> No such file or directory >> >>> *** error retrieving hugepages info *** >> >>> pfring_zc_create_cluster error [No buffer space available] Please >> check your hugetlb configuration >> >>> >> >>> % mount | grep huge >> >>> cgroup on /sys/fs/cgroup/hugetlb type cgroup >> (rw,nosuid,nodev,noexec,relatime,hugetlb) >> >>> hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel) >> >>> nodev on /mnt/huge type hugetlbfs (rw,relatime,seclabel) >> >>> >> >>> Jim >> >>> >> >>> >> >>> >> >>> _______________________________________________ >> >>> Ntop-misc mailing list >> >>> [email protected] <mailto:[email protected]> >> >>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >> > >> > >> > >> >> _______________________________________________ >> >> Ntop-misc mailing list >> >> [email protected] <mailto:[email protected]> >> >> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >> > >> > _______________________________________________ >> > Ntop-misc mailing list >> > [email protected] <mailto:[email protected]> >> > http://listgateway.unipi.it/mailman/listinfo/ntop-misc >> > >> >> >> _______________________________________________ >> Ntop-misc mailing list >> [email protected] <mailto:[email protected]> >> http://listgateway.unipi.it/mailman/listinfo/ntop-misc > > > > _______________________________________________ > Ntop-misc mailing list > [email protected] > http://listgateway.unipi.it/mailman/listinfo/ntop-misc > -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iF4EAREIAAYFAlT5s/AACgkQCGX2wHRYUXRPrAD+L1/j0RaVI0EeKTAcA65YxBQx FcFS+vhEKIYwhADLZEQA/1Yl8sX4YM9Kj17h3/3hI2mANYAWZa0hr14nvVJNgkqI =1XOV -----END PGP SIGNATURE----- _______________________________________________ Ntop-misc mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop-misc
