Some debugging traces related to the general protection fault
strace output:
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
gettimeofday({1457373770, 766718}, NULL) = 0
gettimeofday({1457373770, 769026}, NULL) = 0
gettimeofday({1457373770, 769789}, NULL) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigaction(SIGCHLD, NULL, {SIG_IGN, [CHLD], SA_RESTORER|SA_RESTART,
0x7fd7e30a6d40}, 8) = 0
nanosleep({5, 0}, 0x7fffad633440) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
gettimeofday({1457373775, 805276}, NULL) = 0
gettimeofday({1457373775, 807707}, NULL) = 0
gettimeofday({1457373775, 808547}, NULL) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigaction(SIGCHLD, NULL, {SIG_IGN, [CHLD], SA_RESTORER|SA_RESTART,
0x7fd7e30a6d40}, 8) = 0
nanosleep({5, 0}, 0x7fffad633440) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
gettimeofday({1457373780, 844569}, NULL) = 0
gettimeofday({1457373780, 847055}, NULL) = 0
gettimeofday({1457373780, 847872}, NULL) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigaction(SIGCHLD, NULL, {SIG_IGN, [CHLD], SA_RESTORER|SA_RESTART,
0x7fd7e30a6d40}, 8) = 0
nanosleep({5, 0}, <unfinished ...>
+++ killed by SIGSEGV (core dumped) +++
Segmentation fault (core dumped)
gdb output:
(gdb) file ntopng
Reading symbols from ntopng...done.
(gdb) r
Starting program: /usr/local/bin/ntopng
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
07/Mar/2016 11:34:09 [Ntop.cpp:980] Setting local networks to 127.0.0.0/8
07/Mar/2016 11:34:09 [Redis.cpp:106] Successfully connected to redis
127.0.0.1:6379@0
07/Mar/2016 11:34:09 [NtopPro.cpp:116] [LICENSE] Read license from Redis
[A0400B51C12832343F2A3EF65F4A570014860609709C4EEF2D]
07/Mar/2016 11:34:09 [PcapInterface.cpp:85] Reading packets from interface
eth0...
07/Mar/2016 11:34:09 [Ntop.cpp:1199] Registered interface eth0 [id: 0]
07/Mar/2016 11:34:09 [PcapInterface.cpp:85] Reading packets from interface lo...
07/Mar/2016 11:34:09 [Ntop.cpp:1199] Registered interface lo [id: 1]
07/Mar/2016 11:34:09 [PcapInterface.cpp:85] Reading packets from interface
eth1...
07/Mar/2016 11:34:09 [Ntop.cpp:1199] Registered interface eth1 [id: 6]
07/Mar/2016 11:34:09 [Ntop.cpp:1212] Registered interface view eth0 [id: 0]
07/Mar/2016 11:34:09 [Ntop.cpp:1212] Registered interface view lo [id: 1]
07/Mar/2016 11:34:09 [Ntop.cpp:1212] Registered interface view eth1 [id: 6]
07/Mar/2016 11:34:09 [Utils.cpp:305] User changed to nobody
07/Mar/2016 11:34:09 [main.cpp:258] PID stored in file /var/tmp/ntopng.pid
07/Mar/2016 11:34:09 [HTTPserver.cpp:460] Please read
https://github.com/ntop/ntopng/blob/dev/doc/README.SSL if you want to enable
SSL.
[New Thread 0x7fffe8fb9700 (LWP 25658)]
[New Thread 0x7fffe87b8700 (LWP 25659)]
[New Thread 0x7fffe7fb7700 (LWP 25660)]
[New Thread 0x7fffe77b6700 (LWP 25661)]
[New Thread 0x7fffe6fb5700 (LWP 25662)]
[New Thread 0x7fffe67b4700 (LWP 25663)]
07/Mar/2016 11:34:09 [HTTPserver.cpp:503] Web server dirs
[/usr/share/ntopng/httpdocs][/usr/share/ntopng/scripts]
07/Mar/2016 11:34:09 [HTTPserver.cpp:506] HTTP server listening on port 3000
07/Mar/2016 11:34:09 [main.cpp:295] Working directory: /var/tmp/ntopng
07/Mar/2016 11:34:09 [main.cpp:297] Scripts/HTML pages directory:
/usr/share/ntopng
07/Mar/2016 11:34:09 [Ntop.cpp:258] Welcome to ntopng x86_64 v.2.3.160303 - (C)
1998-16 ntop.org
07/Mar/2016 11:34:09 [Ntop.cpp:263] Built on Ubuntu 14.04.3 LTS
07/Mar/2016 11:34:09 [PeriodicActivities.cpp:53] Started periodic activities
loop...
[New Thread 0x7fffe5fb3700 (LWP 25664)]
[New Thread 0x7fffe57b2700 (LWP 25665)]
[New Thread 0x7fffe4fb1700 (LWP 25666)]
[New Thread 0x7fffc7fff700 (LWP 25667)]
07/Mar/2016 11:34:09 [RuntimePrefs.cpp:34] Dumping alerts into syslog
07/Mar/2016 11:34:09 [NtopPro.cpp:233] [LICENSE] ntopng systemId:
0DA9358100600F2D
07/Mar/2016 11:34:09 [NtopPro.cpp:244] [LICENSE] ntopng license:
A0400B51C12832343F2A3EF65F4A570014860609709C4EEF2D
07/Mar/2016 11:34:09 [NtopPro.cpp:265] [LICENSE] Maintenance is available until
Thu Feb 2 11:42:50 2017 [332 days left]
07/Mar/2016 11:34:09 [Ntop.cpp:514] Adding 127.0.0.0/8 as IPv4 local network
for lo
07/Mar/2016 11:34:09 [Ntop.cpp:514] Adding 10.99.1.128/26 as IPv4 local network
for eth0
07/Mar/2016 11:34:09 [Ntop.cpp:544] Adding ::1/128 as IPv6 local network for lo
07/Mar/2016 11:34:09 [Ntop.cpp:544] Adding fe80::86f:fbff:fea6:2f7f/64 as IPv6
local network for eth0
07/Mar/2016 11:34:09 [Ntop.cpp:544] Adding fe80::b883:e5ff:fe24:82ff/64 as IPv6
local network for eth1
[New Thread 0x7fffc77fe700 (LWP 25668)]
07/Mar/2016 11:34:09 [NetworkInterface.cpp:1431] Started packet polling on
interface eth0 [id: 0]...
[New Thread 0x7fffc6ffd700 (LWP 25669)]
07/Mar/2016 11:34:09 [NetworkInterface.cpp:1431] Started packet polling on
interface lo [id: 1]...
[New Thread 0x7fffc67fc700 (LWP 25670)]
07/Mar/2016 11:34:09 [NetworkInterface.cpp:1431] Started packet polling on
interface eth1 [id: 6]...
[New Thread 0x7fffc5ffb700 (LWP 25671)]
[New Thread 0x7fffc57fa700 (LWP 25672)]
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffc67fc700 (LWP 25670)]
__GI___libc_free (mem=0xe82bdb318667aec2) at malloc.c:2929
2929 malloc.c: No such file or directory.
(gdb) bt full
#0 __GI___libc_free (mem=0xe82bdb318667aec2) at malloc.c:2929
ar_ptr = <optimized out>
p = <optimized out>
hook = 0x0
#1 0x000000000045317e in free_wrapper (freeable=0xe82bdb318667aec2) at
src/NetworkInterface.cpp:40
No locals.
#2 0x0000000000499cca in ndpi_free_flow (flow=0x7fffac986ab0) at
ndpi_main.c:4422
No locals.
#3 0x0000000000416257 in Flow::deleteFlowMemory (this=0x7fffbfcb8100) at
src/Flow.cpp:145
No locals.
#4 0x0000000000456457 in NetworkInterface::processPacket (this=0x28b3ba0,
when=0x38fbdc0, time=1457376054494, eth=0x38fbf50, vlan_id=0,
iph=0x38fbf5e, ip6=0x0, ipsize=45, rawsize=59, h=0x38fbdc0,
packet=0x38fbf50 "\004\030\326\361\266\265\004\030\326\361\263\363\b",
a_shaper_id=0x7fffc67fbaa8, b_shaper_id=0x7fffc67fbaac,
ndpiProtocol=0x7fffc67fbaa6) at src/NetworkInterface.cpp:883
dump_is_unknown = false
src2dst_direction = true
l4_proto = 17 '\021'
flow = 0x7fffbfcb8100
src_port = 31754
l4 = 0x38fbf72 "\n|"
dst_ip = {addr = {ipVersion = 4 '\004', localHost = 0 '\000', privateIP
= 0 '\000', multicastIP = 0 '\000', broadcastIP = 0 '\000',
notUsed = 0 '\000', ipType = {ipv6 = {u6_addr = {u6_addr8 =
"\004\002\002\002", '\000' <repeats 11 times>, u6_addr16 = {516, 514, 0, 0,
0, 0, 0, 0}, u6_addr32 = {33686020, 0, 0, 0}}}, ipv4 =
33686020}}, ip_key = 67240450}
l4_packet_len = 25
tcp_flags = 0 '\000'
payload = 0x38fbf7a "\002\003\202\001\001"
is_fragment = false
new_flow = true
eth_dst = 0x38fbf50 "\004\030\326\361\266\265\004\030\326\361\263\363\b"
tcph = 0x0
eth_src = 0x38fbf56 "\004\030\326\361\263\363\b"
dst_port = 13568
pass_verdict = true
src_ip = {addr = {ipVersion = 4 '\004', localHost = 0 '\000', privateIP
= 0 '\000', multicastIP = 0 '\000', broadcastIP = 0 '\000',
notUsed = 0 '\000', ipType = {ipv6 = {u6_addr = {u6_addr8 =
"\300\256\006>", '\000' <repeats 11 times>, u6_addr16 = {44736, 15878, 0, 0,
0, 0, 0, 0}, u6_addr32 = {1040625344, 0, 0, 0}}}, ipv4 =
1040625344}}, ip_key = 3232630334}
payload_len = 17
udph = 0x38fbf72
ip = 0x38fbf5e "E"
#5 0x0000000000457b04 in NetworkInterface::dissectPacket (this=0x28b3ba0,
h=0x38fbdc0,
packet=0x38fbf50 "\004\030\326\361\266\265\004\030\326\361\263\363\b",
a_shaper_id=0x7fffc67fbaa8, b_shaper_id=0x7fffc67fbaac,
ndpiProtocol=0x7fffc67fbaa6) at src/NetworkInterface.cpp:1299
frag_off = 16384
iph = 0x38fbf5e
ip6 = 0x0
srcHost = 0x7fffbc290e50
dstHost = 0x38fbdc0
ethernet = 0x38fbf50
dummy_ethernet = {h_dest = "\000\307\177\306\377\177", h_source =
"\000\000\002\234\272", <incomplete sequence \367>, h_proto = 32767}
time = 1457376054494
---Type <return> to continue, or q <return> to quit---
eth_type = 2048
res = 1000
vlan_id = 0
eth_offset = 0
lasttime = 1457376054494
ip_offset = 14
null_type = 1
pcap_datalink_type = 1
pass_verdict = true
#6 0x00000000004685aa in packetPollLoop (ptr=0x28b3ba0) at
src/PcapInterface.cpp:187
a = 0
b = 0
p = 0
pkt = 0x38fbf50 "\004\030\326\361\266\265\004\030\326\361\263\363\b"
hdr = 0x38fbdc0
rc = 1
iface = 0x28b3ba0
pd = 0x38fbbc0
pcap_list = 0x0
#7 0x00007ffff6af4182 in start_thread (arg=0x7fffc67fc700) at
pthread_create.c:312
__res = <optimized out>
pd = 0x7fffc67fc700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140736523650816,
-6549973915309047318, 1, 0, 140736523651520, 140736523650816,
6550025590167962090, 6549988901613994474}, mask_was_saved =
0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0,
cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
pagesize_m1 = <optimized out>
sp = <optimized out>
freesize = <optimized out>
__PRETTY_FUNCTION__ = "start_thread"
#8 0x00007ffff2a3147d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.
----- Original Message -----
From: "Jerome Backmeyer" <[email protected]>
To: [email protected]
Sent: Thursday, March 3, 2016 4:41:24 PM
Subject: Re: [Ntop] (no subject)
Ack, spoke too soon...
kernel: [89088.694285] traps: ntopng[26383] general protection ip:7f545462ae0c
sp:7f54417f96e8 error:0 in libc-2.19.so[7f54545a8000+1bb000]
12 minutes 31 seconds of runtime
Jerome
----- Original Message -----
From: "Jerome Backmeyer" <[email protected]>
To: [email protected]
Sent: Thursday, March 3, 2016 4:38:56 PM
Subject: Re: [Ntop] (no subject)
Not really but a bit of google goes a long way. I've removed and rebuilt the
.ko for the current kernel
modinfo /var/lib/dkms/pfring/6.3.0/3.13.0-79-generic/x86_64/module/pf_ring.ko
filename:
/var/lib/dkms/pfring/6.3.0/3.13.0-79-generic/x86_64/module/pf_ring.ko
alias: net-pf-27
description: Packet capture acceleration and analysis
author: ntop.org
license: GPL
srcversion: 5E66F2A01976FA528D3EA38
depends:
vermagic: 3.13.0-79-generic SMP mod_unload modversions
parm: min_num_slots:Min number of ring slots (uint)
parm: perfect_rules_hash_size:Perfect rules hash size (uint)
parm: transparent_mode:(deprecated) (uint)
parm: enable_debug:Set to 1 to enable PF_RING debug tracing into the
syslog (uint)
parm: enable_tx_capture:Set to 1 to capture outgoing packets (uint)
parm: enable_frag_coherence:Set to 1 to handle fragments (flow
coherence) in clusters (uint)
parm: enable_ip_defrag:Set to 1 to enable IP defragmentation(only rx
traffic is defragmentead) (uint)
parm: quick_mode:Set to 1 to run at full speed but with upto one
socket per interface (uint)
The module error is gone. I restart ntopng, currently at 12 minutes of runtime
and counting without a general protection error.
That looks very hopeful...
Thanks!
----- Original Message -----
From: "Alfredo Cardigliano" <[email protected]>
To: [email protected]
Sent: Thursday, March 3, 2016 4:09:32 PM
Subject: Re: [Ntop] (no subject)
Are you familiar with dkms? Could you try rebuilding it? Is seems to be tied to
another kernel version.
Alfredo
> On 04 Mar 2016, at 00:04, Jerome Backmeyer <[email protected]> wrote:
>
> Well that's interesting
>
> modinfo /lib/modules/3.13.0-79-generic/updates/dkms/pf_ring.ko
> filename: /lib/modules/3.13.0-79-generic/updates/dkms/pf_ring.ko
> alias: net-pf-27
> description: Packet capture acceleration and analysis
> author: ntop.org
> license: GPL
> srcversion: 5E66F2A01976FA528D3EA38
> depends:
> vermagic: 3.13.0-77-generic SMP mod_unload modversions
> parm: min_num_slots:Min number of ring slots (uint)
> parm: perfect_rules_hash_size:Perfect rules hash size (uint)
> parm: transparent_mode:(deprecated) (uint)
> parm: enable_debug:Set to 1 to enable PF_RING debug tracing into
> the syslog (uint)
> parm: enable_tx_capture:Set to 1 to capture outgoing packets (uint)
> parm: enable_frag_coherence:Set to 1 to handle fragments (flow
> coherence) in clusters (uint)
> parm: enable_ip_defrag:Set to 1 to enable IP defragmentation(only
> rx traffic is defragmentead) (uint)
> parm: quick_mode:Set to 1 to run at full speed but with upto one
> socket per interface (uint)
>
>
> ----- Original Message -----
> From: "Alfredo Cardigliano" <[email protected]>
> To: [email protected]
> Sent: Thursday, March 3, 2016 3:48:55 PM
> Subject: Re: [Ntop] (no subject)
>
> Please locate the module you have installed with:
>
> $ updatedb && locate pf_ring.ko
>
> and provide us the output of:
>
> $ modinfo <path>/pf_ring.ko
>
> It should be:
>
> $ modinfo /lib/modules/$(uname -r)/kernel/net/pf_ring/pf_ring.ko
>
> Alfredo
>
>> On 03 Mar 2016, at 23:38, Jerome Backmeyer <[email protected]> wrote:
>>
>> Hi Alfredo
>>
>> I installed using the apt repo packages from
>> http://packages.ntop.org/apt/14.04/ x64/
>> paravirtualized ubuntu server version 14.04, running on citrix Xen Server 6.5
>>
>>
>> Package: pfring
>> Versions:
>> 6.3.0-476 (/var/lib/apt/lists/packages.ntop.org_apt_14.04_x64_Packages)
>> Description Language:
>> File:
>> /var/lib/apt/lists/packages.ntop.org_apt_14.04_x64_Packages
>> MD5: f892cc42e50eb07786ecb03073b119f4
>>
>> 6.3.0-473 (/var/lib/dpkg/status)
>> Description Language:
>> File: /var/lib/dpkg/status
>> MD5: a532d67553054702ed5d4bd58bd675f6
>>
>>
>> Reverse Depends:
>> ntopng,pfring 6.3.0-473
>> n2disk,pfring 6.3.0-476
>> ntopng,pfring 6.3.0-476
>> nprobe,pfring 6.3.0-476
>> Dependencies:
>> 6.3.0-476 - pfring-dkms (5 6.3.0)
>> 6.3.0-473 - pfring-dkms (5 6.3.0)
>>
>>
>> Any guidance would be very appreciated.
>>
>> Thanks
>>
>>
>> ----- Original Message -----
>> From: "Alfredo Cardigliano" <[email protected]>
>> To: [email protected]
>> Sent: Thursday, March 3, 2016 3:22:54 PM
>> Subject: Re: [Ntop] (no subject)
>>
>> Hi Jerome
>> how did you install pf_ring on this machine? Please provide more info about
>> OS, installation method, version, etc.
>>
>> Alfredo
>>
>>> On 29 Feb 2016, at 19:16, Jerome Backmeyer <[email protected]> wrote:
>>>
>>> Yeah, I suspect there is something off with my setup, but I can't figure
>>> what.
>>>
>>> It is running on a Xen Server virtual machine, not usre if that would have
>>> any impact. If I setup a nProbe and start ntopng with -i zmq then it runs
>>> fine, but if it's directly capturing it crashes with the kernel general
>>> protection fault.
>>>
>>> I also get this error: kernel: [86211.917173] pf_ring: disagrees about
>>> version of symbol module_layout
>>>
>>>
>>>
>>> ----- Original Message -----
>>> From: "Thomas Leathley" <[email protected]>
>>> To: [email protected]
>>> Sent: Friday, February 26, 2016 5:26:53 PM
>>> Subject: Re: [Ntop] (no subject)
>>>
>>> I have no issues with a fresh install on 14.04.03 and all the latest
>>> apt-get updates on the whole stack.
>>>
>>> On Fri, Feb 26, 2016 at 3:35 PM, Jerome Backmeyer < [email protected] >
>>> wrote:
>>>
>>>
>>> Hi,
>>>
>>> I just got ntopng running, but I can't seem to keep it up for more than a
>>> few minutes. It'll start I can log into the web interface, then then it
>>> dies with this error:
>>> kernel: [86400.616165] traps: ntopng[23260] general protection
>>> ip:7f7d53cf2e0c sp:7f7d417f96e8 error:0 in libc-2.19.so
>>>
>>> I've installed it from packages on ubuntu 14.04.
>>>
>>> Has anyone run into this before or have a guess as to where I've gone wrong?
>>>
>>> Jerome
>>> _______________________________________________
>>> Ntop mailing list
>>> [email protected]
>>> http://listgateway.unipi.it/mailman/listinfo/ntop
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Ntop mailing list
>>> [email protected]
>>> http://listgateway.unipi.it/mailman/listinfo/ntop
>>>
>>> _______________________________________________
>>> Ntop mailing list
>>> [email protected]
>>> http://listgateway.unipi.it/mailman/listinfo/ntop
>>
>>
>> _______________________________________________
>> Ntop mailing list
>> [email protected]
>> http://listgateway.unipi.it/mailman/listinfo/ntop
>> _______________________________________________
>> Ntop mailing list
>> [email protected]
>> http://listgateway.unipi.it/mailman/listinfo/ntop
>
>
> _______________________________________________
> Ntop mailing list
> [email protected]
> http://listgateway.unipi.it/mailman/listinfo/ntop
> _______________________________________________
> Ntop mailing list
> [email protected]
> http://listgateway.unipi.it/mailman/listinfo/ntop
_______________________________________________
Ntop mailing list
[email protected]
http://listgateway.unipi.it/mailman/listinfo/ntop
_______________________________________________
Ntop mailing list
[email protected]
http://listgateway.unipi.it/mailman/listinfo/ntop
_______________________________________________
Ntop mailing list
[email protected]
http://listgateway.unipi.it/mailman/listinfo/ntop
_______________________________________________
Ntop mailing list
[email protected]
http://listgateway.unipi.it/mailman/listinfo/ntop
