Jerome thanks for reporting the issue but I would appreciated if you can send me ([email protected] <mailto:[email protected]>) a tar file containing both the ntopng binary and the core dump so I can analyse to as I do not see any major issue with the trace you sent
Thanks Luca > On 07 Mar 2016, at 19:43, Jerome Backmeyer <[email protected]> wrote: > > Some debugging traces related to the general protection fault > > > strace output: > > rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 > gettimeofday({1457373770, 766718}, NULL) = 0 > gettimeofday({1457373770, 769026}, NULL) = 0 > gettimeofday({1457373770, 769789}, NULL) = 0 > rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0 > rt_sigaction(SIGCHLD, NULL, {SIG_IGN, [CHLD], SA_RESTORER|SA_RESTART, > 0x7fd7e30a6d40}, 8) = 0 > nanosleep({5, 0}, 0x7fffad633440) = 0 > rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 > gettimeofday({1457373775, 805276}, NULL) = 0 > gettimeofday({1457373775, 807707}, NULL) = 0 > gettimeofday({1457373775, 808547}, NULL) = 0 > rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0 > rt_sigaction(SIGCHLD, NULL, {SIG_IGN, [CHLD], SA_RESTORER|SA_RESTART, > 0x7fd7e30a6d40}, 8) = 0 > nanosleep({5, 0}, 0x7fffad633440) = 0 > rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 > gettimeofday({1457373780, 844569}, NULL) = 0 > gettimeofday({1457373780, 847055}, NULL) = 0 > gettimeofday({1457373780, 847872}, NULL) = 0 > rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0 > rt_sigaction(SIGCHLD, NULL, {SIG_IGN, [CHLD], SA_RESTORER|SA_RESTART, > 0x7fd7e30a6d40}, 8) = 0 > nanosleep({5, 0}, <unfinished ...> > +++ killed by SIGSEGV (core dumped) +++ > Segmentation fault (core dumped) > > > gdb output: > > (gdb) file ntopng > Reading symbols from ntopng...done. > (gdb) r > Starting program: /usr/local/bin/ntopng > [Thread debugging using libthread_db enabled] > Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". > 07/Mar/2016 11:34:09 [Ntop.cpp:980] Setting local networks to 127.0.0.0/8 > 07/Mar/2016 11:34:09 [Redis.cpp:106] Successfully connected to redis > 127.0.0.1:6379@0 > 07/Mar/2016 11:34:09 [NtopPro.cpp:116] [LICENSE] Read license from Redis > [A0400B51C12832343F2A3EF65F4A570014860609709C4EEF2D] > 07/Mar/2016 11:34:09 [PcapInterface.cpp:85] Reading packets from interface > eth0... > 07/Mar/2016 11:34:09 [Ntop.cpp:1199] Registered interface eth0 [id: 0] > 07/Mar/2016 11:34:09 [PcapInterface.cpp:85] Reading packets from interface > lo... > 07/Mar/2016 11:34:09 [Ntop.cpp:1199] Registered interface lo [id: 1] > 07/Mar/2016 11:34:09 [PcapInterface.cpp:85] Reading packets from interface > eth1... > 07/Mar/2016 11:34:09 [Ntop.cpp:1199] Registered interface eth1 [id: 6] > 07/Mar/2016 11:34:09 [Ntop.cpp:1212] Registered interface view eth0 [id: 0] > 07/Mar/2016 11:34:09 [Ntop.cpp:1212] Registered interface view lo [id: 1] > 07/Mar/2016 11:34:09 [Ntop.cpp:1212] Registered interface view eth1 [id: 6] > 07/Mar/2016 11:34:09 [Utils.cpp:305] User changed to nobody > 07/Mar/2016 11:34:09 [main.cpp:258] PID stored in file /var/tmp/ntopng.pid > 07/Mar/2016 11:34:09 [HTTPserver.cpp:460] Please read > https://github.com/ntop/ntopng/blob/dev/doc/README.SSL if you want to enable > SSL. > [New Thread 0x7fffe8fb9700 (LWP 25658)] > [New Thread 0x7fffe87b8700 (LWP 25659)] > [New Thread 0x7fffe7fb7700 (LWP 25660)] > [New Thread 0x7fffe77b6700 (LWP 25661)] > [New Thread 0x7fffe6fb5700 (LWP 25662)] > [New Thread 0x7fffe67b4700 (LWP 25663)] > 07/Mar/2016 11:34:09 [HTTPserver.cpp:503] Web server dirs > [/usr/share/ntopng/httpdocs][/usr/share/ntopng/scripts] > 07/Mar/2016 11:34:09 [HTTPserver.cpp:506] HTTP server listening on port 3000 > 07/Mar/2016 11:34:09 [main.cpp:295] Working directory: /var/tmp/ntopng > 07/Mar/2016 11:34:09 [main.cpp:297] Scripts/HTML pages directory: > /usr/share/ntopng > 07/Mar/2016 11:34:09 [Ntop.cpp:258] Welcome to ntopng x86_64 v.2.3.160303 - > (C) 1998-16 ntop.org > 07/Mar/2016 11:34:09 [Ntop.cpp:263] Built on Ubuntu 14.04.3 LTS > 07/Mar/2016 11:34:09 [PeriodicActivities.cpp:53] Started periodic activities > loop... > [New Thread 0x7fffe5fb3700 (LWP 25664)] > [New Thread 0x7fffe57b2700 (LWP 25665)] > [New Thread 0x7fffe4fb1700 (LWP 25666)] > [New Thread 0x7fffc7fff700 (LWP 25667)] > 07/Mar/2016 11:34:09 [RuntimePrefs.cpp:34] Dumping alerts into syslog > 07/Mar/2016 11:34:09 [NtopPro.cpp:233] [LICENSE] ntopng systemId: > 0DA9358100600F2D > 07/Mar/2016 11:34:09 [NtopPro.cpp:244] [LICENSE] ntopng license: > A0400B51C12832343F2A3EF65F4A570014860609709C4EEF2D > 07/Mar/2016 11:34:09 [NtopPro.cpp:265] [LICENSE] Maintenance is available > until Thu Feb 2 11:42:50 2017 [332 days left] > 07/Mar/2016 11:34:09 [Ntop.cpp:514] Adding 127.0.0.0/8 as IPv4 local network > for lo > 07/Mar/2016 11:34:09 [Ntop.cpp:514] Adding 10.99.1.128/26 as IPv4 local > network for eth0 > 07/Mar/2016 11:34:09 [Ntop.cpp:544] Adding ::1/128 as IPv6 local network for > lo > 07/Mar/2016 11:34:09 [Ntop.cpp:544] Adding fe80::86f:fbff:fea6:2f7f/64 as > IPv6 local network for eth0 > 07/Mar/2016 11:34:09 [Ntop.cpp:544] Adding fe80::b883:e5ff:fe24:82ff/64 as > IPv6 local network for eth1 > [New Thread 0x7fffc77fe700 (LWP 25668)] > 07/Mar/2016 11:34:09 [NetworkInterface.cpp:1431] Started packet polling on > interface eth0 [id: 0]... > [New Thread 0x7fffc6ffd700 (LWP 25669)] > 07/Mar/2016 11:34:09 [NetworkInterface.cpp:1431] Started packet polling on > interface lo [id: 1]... > [New Thread 0x7fffc67fc700 (LWP 25670)] > 07/Mar/2016 11:34:09 [NetworkInterface.cpp:1431] Started packet polling on > interface eth1 [id: 6]... > [New Thread 0x7fffc5ffb700 (LWP 25671)] > [New Thread 0x7fffc57fa700 (LWP 25672)] > > Program received signal SIGSEGV, Segmentation fault. > [Switching to Thread 0x7fffc67fc700 (LWP 25670)] > __GI___libc_free (mem=0xe82bdb318667aec2) at malloc.c:2929 > 2929 malloc.c: No such file or directory. > (gdb) bt full > #0 __GI___libc_free (mem=0xe82bdb318667aec2) at malloc.c:2929 > ar_ptr = <optimized out> > p = <optimized out> > hook = 0x0 > #1 0x000000000045317e in free_wrapper (freeable=0xe82bdb318667aec2) at > src/NetworkInterface.cpp:40 > No locals. > #2 0x0000000000499cca in ndpi_free_flow (flow=0x7fffac986ab0) at > ndpi_main.c:4422 > No locals. > #3 0x0000000000416257 in Flow::deleteFlowMemory (this=0x7fffbfcb8100) at > src/Flow.cpp:145 > No locals. > #4 0x0000000000456457 in NetworkInterface::processPacket (this=0x28b3ba0, > when=0x38fbdc0, time=1457376054494, eth=0x38fbf50, vlan_id=0, > iph=0x38fbf5e, ip6=0x0, ipsize=45, rawsize=59, h=0x38fbdc0, > packet=0x38fbf50 "\004\030\326\361\266\265\004\030\326\361\263\363\b", > a_shaper_id=0x7fffc67fbaa8, b_shaper_id=0x7fffc67fbaac, > ndpiProtocol=0x7fffc67fbaa6) at src/NetworkInterface.cpp:883 > dump_is_unknown = false > src2dst_direction = true > l4_proto = 17 '\021' > flow = 0x7fffbfcb8100 > src_port = 31754 > l4 = 0x38fbf72 "\n|" > dst_ip = {addr = {ipVersion = 4 '\004', localHost = 0 '\000', > privateIP = 0 '\000', multicastIP = 0 '\000', broadcastIP = 0 '\000', > notUsed = 0 '\000', ipType = {ipv6 = {u6_addr = {u6_addr8 = > "\004\002\002\002", '\000' <repeats 11 times>, u6_addr16 = {516, 514, 0, 0, > 0, 0, 0, 0}, u6_addr32 = {33686020, 0, 0, 0}}}, ipv4 = > 33686020}}, ip_key = 67240450} > l4_packet_len = 25 > tcp_flags = 0 '\000' > payload = 0x38fbf7a "\002\003\202\001\001" > is_fragment = false > new_flow = true > eth_dst = 0x38fbf50 > "\004\030\326\361\266\265\004\030\326\361\263\363\b" > tcph = 0x0 > eth_src = 0x38fbf56 "\004\030\326\361\263\363\b" > dst_port = 13568 > pass_verdict = true > src_ip = {addr = {ipVersion = 4 '\004', localHost = 0 '\000', > privateIP = 0 '\000', multicastIP = 0 '\000', broadcastIP = 0 '\000', > notUsed = 0 '\000', ipType = {ipv6 = {u6_addr = {u6_addr8 = > "\300\256\006>", '\000' <repeats 11 times>, u6_addr16 = {44736, 15878, 0, 0, > 0, 0, 0, 0}, u6_addr32 = {1040625344, 0, 0, 0}}}, ipv4 = > 1040625344}}, ip_key = 3232630334} > payload_len = 17 > udph = 0x38fbf72 > ip = 0x38fbf5e "E" > #5 0x0000000000457b04 in NetworkInterface::dissectPacket (this=0x28b3ba0, > h=0x38fbdc0, > packet=0x38fbf50 "\004\030\326\361\266\265\004\030\326\361\263\363\b", > a_shaper_id=0x7fffc67fbaa8, b_shaper_id=0x7fffc67fbaac, > ndpiProtocol=0x7fffc67fbaa6) at src/NetworkInterface.cpp:1299 > frag_off = 16384 > iph = 0x38fbf5e > ip6 = 0x0 > srcHost = 0x7fffbc290e50 > dstHost = 0x38fbdc0 > ethernet = 0x38fbf50 > dummy_ethernet = {h_dest = "\000\307\177\306\377\177", h_source = > "\000\000\002\234\272", <incomplete sequence \367>, h_proto = 32767} > time = 1457376054494 > ---Type <return> to continue, or q <return> to quit--- > eth_type = 2048 > res = 1000 > vlan_id = 0 > eth_offset = 0 > lasttime = 1457376054494 > ip_offset = 14 > null_type = 1 > pcap_datalink_type = 1 > pass_verdict = true > #6 0x00000000004685aa in packetPollLoop (ptr=0x28b3ba0) at > src/PcapInterface.cpp:187 > a = 0 > b = 0 > p = 0 > pkt = 0x38fbf50 "\004\030\326\361\266\265\004\030\326\361\263\363\b" > hdr = 0x38fbdc0 > rc = 1 > iface = 0x28b3ba0 > pd = 0x38fbbc0 > pcap_list = 0x0 > #7 0x00007ffff6af4182 in start_thread (arg=0x7fffc67fc700) at > pthread_create.c:312 > __res = <optimized out> > pd = 0x7fffc67fc700 > now = <optimized out> > unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140736523650816, > -6549973915309047318, 1, 0, 140736523651520, 140736523650816, > 6550025590167962090, 6549988901613994474}, mask_was_saved = > 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, > cleanup = 0x0, canceltype = 0}}} > not_first_call = <optimized out> > pagesize_m1 = <optimized out> > sp = <optimized out> > freesize = <optimized out> > __PRETTY_FUNCTION__ = "start_thread" > #8 0x00007ffff2a3147d in clone () at > ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 > No locals. > > > > > ----- Original Message ----- > From: "Jerome Backmeyer" <[email protected]> > To: [email protected] > Sent: Thursday, March 3, 2016 4:41:24 PM > Subject: Re: [Ntop] (no subject) > > Ack, spoke too soon... > > kernel: [89088.694285] traps: ntopng[26383] general protection > ip:7f545462ae0c sp:7f54417f96e8 error:0 in libc-2.19.so[7f54545a8000+1bb000] > > 12 minutes 31 seconds of runtime > > > Jerome > > ----- Original Message ----- > From: "Jerome Backmeyer" <[email protected]> > To: [email protected] > Sent: Thursday, March 3, 2016 4:38:56 PM > Subject: Re: [Ntop] (no subject) > > Not really but a bit of google goes a long way. I've removed and rebuilt the > .ko for the current kernel > > modinfo /var/lib/dkms/pfring/6.3.0/3.13.0-79-generic/x86_64/module/pf_ring.ko > filename: > /var/lib/dkms/pfring/6.3.0/3.13.0-79-generic/x86_64/module/pf_ring.ko > alias: net-pf-27 > description: Packet capture acceleration and analysis > author: ntop.org > license: GPL > srcversion: 5E66F2A01976FA528D3EA38 > depends: > vermagic: 3.13.0-79-generic SMP mod_unload modversions > parm: min_num_slots:Min number of ring slots (uint) > parm: perfect_rules_hash_size:Perfect rules hash size (uint) > parm: transparent_mode:(deprecated) (uint) > parm: enable_debug:Set to 1 to enable PF_RING debug tracing into > the syslog (uint) > parm: enable_tx_capture:Set to 1 to capture outgoing packets (uint) > parm: enable_frag_coherence:Set to 1 to handle fragments (flow > coherence) in clusters (uint) > parm: enable_ip_defrag:Set to 1 to enable IP defragmentation(only > rx traffic is defragmentead) (uint) > parm: quick_mode:Set to 1 to run at full speed but with upto one > socket per interface (uint) > > > > The module error is gone. I restart ntopng, currently at 12 minutes of > runtime and counting without a general protection error. > > That looks very hopeful... > > Thanks! > > ----- Original Message ----- > From: "Alfredo Cardigliano" <[email protected]> > To: [email protected] > Sent: Thursday, March 3, 2016 4:09:32 PM > Subject: Re: [Ntop] (no subject) > > Are you familiar with dkms? Could you try rebuilding it? Is seems to be tied > to another kernel version. > > Alfredo > >> On 04 Mar 2016, at 00:04, Jerome Backmeyer <[email protected]> wrote: >> >> Well that's interesting >> >> modinfo /lib/modules/3.13.0-79-generic/updates/dkms/pf_ring.ko >> filename: /lib/modules/3.13.0-79-generic/updates/dkms/pf_ring.ko >> alias: net-pf-27 >> description: Packet capture acceleration and analysis >> author: ntop.org >> license: GPL >> srcversion: 5E66F2A01976FA528D3EA38 >> depends: >> vermagic: 3.13.0-77-generic SMP mod_unload modversions >> parm: min_num_slots:Min number of ring slots (uint) >> parm: perfect_rules_hash_size:Perfect rules hash size (uint) >> parm: transparent_mode:(deprecated) (uint) >> parm: enable_debug:Set to 1 to enable PF_RING debug tracing into >> the syslog (uint) >> parm: enable_tx_capture:Set to 1 to capture outgoing packets (uint) >> parm: enable_frag_coherence:Set to 1 to handle fragments (flow >> coherence) in clusters (uint) >> parm: enable_ip_defrag:Set to 1 to enable IP defragmentation(only >> rx traffic is defragmentead) (uint) >> parm: quick_mode:Set to 1 to run at full speed but with upto one >> socket per interface (uint) >> >> >> ----- Original Message ----- >> From: "Alfredo Cardigliano" <[email protected]> >> To: [email protected] >> Sent: Thursday, March 3, 2016 3:48:55 PM >> Subject: Re: [Ntop] (no subject) >> >> Please locate the module you have installed with: >> >> $ updatedb && locate pf_ring.ko >> >> and provide us the output of: >> >> $ modinfo <path>/pf_ring.ko >> >> It should be: >> >> $ modinfo /lib/modules/$(uname -r)/kernel/net/pf_ring/pf_ring.ko >> >> Alfredo >> >>> On 03 Mar 2016, at 23:38, Jerome Backmeyer <[email protected]> wrote: >>> >>> Hi Alfredo >>> >>> I installed using the apt repo packages from >>> http://packages.ntop.org/apt/14.04/ x64/ >>> paravirtualized ubuntu server version 14.04, running on citrix Xen Server >>> 6.5 >>> >>> >>> Package: pfring >>> Versions: >>> 6.3.0-476 (/var/lib/apt/lists/packages.ntop.org_apt_14.04_x64_Packages) >>> Description Language: >>> File: >>> /var/lib/apt/lists/packages.ntop.org_apt_14.04_x64_Packages >>> MD5: f892cc42e50eb07786ecb03073b119f4 >>> >>> 6.3.0-473 (/var/lib/dpkg/status) >>> Description Language: >>> File: /var/lib/dpkg/status >>> MD5: a532d67553054702ed5d4bd58bd675f6 >>> >>> >>> Reverse Depends: >>> ntopng,pfring 6.3.0-473 >>> n2disk,pfring 6.3.0-476 >>> ntopng,pfring 6.3.0-476 >>> nprobe,pfring 6.3.0-476 >>> Dependencies: >>> 6.3.0-476 - pfring-dkms (5 6.3.0) >>> 6.3.0-473 - pfring-dkms (5 6.3.0) >>> >>> >>> Any guidance would be very appreciated. >>> >>> Thanks >>> >>> >>> ----- Original Message ----- >>> From: "Alfredo Cardigliano" <[email protected]> >>> To: [email protected] >>> Sent: Thursday, March 3, 2016 3:22:54 PM >>> Subject: Re: [Ntop] (no subject) >>> >>> Hi Jerome >>> how did you install pf_ring on this machine? Please provide more info about >>> OS, installation method, version, etc. >>> >>> Alfredo >>> >>>> On 29 Feb 2016, at 19:16, Jerome Backmeyer <[email protected]> wrote: >>>> >>>> Yeah, I suspect there is something off with my setup, but I can't figure >>>> what. >>>> >>>> It is running on a Xen Server virtual machine, not usre if that would have >>>> any impact. If I setup a nProbe and start ntopng with -i zmq then it runs >>>> fine, but if it's directly capturing it crashes with the kernel general >>>> protection fault. >>>> >>>> I also get this error: kernel: [86211.917173] pf_ring: disagrees about >>>> version of symbol module_layout >>>> >>>> >>>> >>>> ----- Original Message ----- >>>> From: "Thomas Leathley" <[email protected]> >>>> To: [email protected] >>>> Sent: Friday, February 26, 2016 5:26:53 PM >>>> Subject: Re: [Ntop] (no subject) >>>> >>>> I have no issues with a fresh install on 14.04.03 and all the latest >>>> apt-get updates on the whole stack. >>>> >>>> On Fri, Feb 26, 2016 at 3:35 PM, Jerome Backmeyer < [email protected] >>>> > wrote: >>>> >>>> >>>> Hi, >>>> >>>> I just got ntopng running, but I can't seem to keep it up for more than a >>>> few minutes. It'll start I can log into the web interface, then then it >>>> dies with this error: >>>> kernel: [86400.616165] traps: ntopng[23260] general protection >>>> ip:7f7d53cf2e0c sp:7f7d417f96e8 error:0 in libc-2.19.so >>>> >>>> I've installed it from packages on ubuntu 14.04. >>>> >>>> Has anyone run into this before or have a guess as to where I've gone >>>> wrong? >>>> >>>> Jerome >>>> _______________________________________________ >>>> Ntop mailing list >>>> [email protected] >>>> http://listgateway.unipi.it/mailman/listinfo/ntop >>>> >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> Ntop mailing list >>>> [email protected] >>>> http://listgateway.unipi.it/mailman/listinfo/ntop >>>> >>>> _______________________________________________ >>>> Ntop mailing list >>>> [email protected] >>>> http://listgateway.unipi.it/mailman/listinfo/ntop >>> >>> >>> _______________________________________________ >>> Ntop mailing list >>> [email protected] >>> http://listgateway.unipi.it/mailman/listinfo/ntop >>> _______________________________________________ >>> Ntop mailing list >>> [email protected] >>> http://listgateway.unipi.it/mailman/listinfo/ntop >> >> >> _______________________________________________ >> Ntop mailing list >> [email protected] >> http://listgateway.unipi.it/mailman/listinfo/ntop >> _______________________________________________ >> Ntop mailing list >> [email protected] >> http://listgateway.unipi.it/mailman/listinfo/ntop > > > _______________________________________________ > Ntop mailing list > [email protected] > http://listgateway.unipi.it/mailman/listinfo/ntop > _______________________________________________ > Ntop mailing list > [email protected] > http://listgateway.unipi.it/mailman/listinfo/ntop > _______________________________________________ > Ntop mailing list > [email protected] > http://listgateway.unipi.it/mailman/listinfo/ntop > _______________________________________________ > Ntop mailing list > [email protected] > http://listgateway.unipi.it/mailman/listinfo/ntop
_______________________________________________ Ntop mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop
