I think that's specious. ntop doesn't read the database into memory. We just do queries against it.
There's two thoughts - either (1) gdbm db has been corrupted (leading ntop to put bad stuff into memory) or (2) something else corrupted the HostTraffic chains. Try --trace-level 6 ... That will give you the file/line where the message is getting generated: Apr 25 09:35:50 tigger devel[15751]: [MSGID0836899] [main:454] Copyright 1998-2005 by Luca Deri <[EMAIL PROTECTED]> Apr 25 09:35:50 tigger devel[15751]: [MSGID0260707] [main:455] Get the freshest ntop from http://www.ntop.org/ Apr 25 09:35:50 tigger devel[15751]: [MSGID0188699] [main:459] NOTE: ntop is running from '/devel/bin' But, since you are seeing both msgs, it's probably the ones in lookupHost() (hash.c around 860). That doesn't say much - it's the general "gee, there is a bad HostTraffic record" info. Because the data in el->ethAddressString (as reported in the 'Error: wrong bucketIdx' msg) is bunk, I lean towards (1). You can try using dumpgdbm (I've posted this before) and/or dnscachePurge (SourceForge). Still, if you delete the cache and rerun and it reappears, remember HostTraffic entries are an array of linked lists: HT[0] HT[1]->HT->HT HT[2]->HT HT[3] Etc. Check the info.html page - it gives you some stuff re the chains: Host/Session counts - Device 0 (eth1) Hash Bucket Size 1.9 KB Actual Hash Size 16384 Stored hosts 39 Bucket List Length [min 1][max 5][avg 1.1] Max host lookup 4 Actual hash size is the number of entries in the array. Bucket list length are the Linked lists. -----Burton -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Robbert Kouprie Sent: Monday, April 25, 2005 9:06 AM To: [email protected] Subject: [Ntop-dev] Data corruption [was: Ntop segfaulting] (fwd) I tracked it down to my 47Mb large dnsCache.db file. When I read its contents into a hash with a Perl script using GDBM_File, the script crashed with either a "read error" or a "malloc error" (depending on which system I ran it on; first one has 4G, second 512M of memory). Maybe we should add some sort of sanity check for the GDBM databases at ntop startup? Robbert ---------- Forwarded message ---------- Date: Mon, 25 Apr 2005 13:47:48 +0200 (CEST) From: Robbert Kouprie <[EMAIL PROTECTED]> Reply-To: [email protected] To: [email protected] Subject: [Ntop-dev] Data corruption [was: Ntop segfaulting] Ok, Looks like ntop was crashing on corrupted data files. After removing addressQueue.db, dnsCache.db and macPrefix.db ntop (current CVS now) runs again happily. The corruption started about 2 weeks ago. ntop began suddenly spewing "bad magic" messages in the logs, and would crash immediately after this message due to a segfault in lookupHost. I was using the CVS version of April 5th, but current CVS would also segfault (in a different place - but that doesn't tell much). So, two options: 1) hardware is causing corruption 2) ntop itself is causing corruption after a while To look for the source of the corruption, I will launch a thorough memory test tonight (did some basic tests already - nothing found). Just for completeness I'll include the log entries. Maybe something abvious jumps out for someone. Regards, Robbert Apr 13 12:18:25 ntop2 ntop[32564]: [MSGID9333267] **WARNING** Error: bad magic number (expected=1968/real=1406) [deviceId=0] Apr 13 12:18:25 ntop2 ntop[32564]: [MSGID9333267] **WARNING** Error: wrong bucketIdx :2C:CD:D4/128.61 (expected=1406/real=0) [deviceId=0] -- Apr 13 12:20:22 ntop2 ntop[7411]: [MSGID9123622] **WARNING** Error: bad magic number [expected=1968/real=9044][deviceId=1] Apr 13 12:20:22 ntop2 kernel: device eth0 left promiscuous mode -- Apr 13 12:26:29 ntop2 ntop[7446]: [MSGID9333267] **WARNING** Error: bad magic number (expected=1968/real=1750) [deviceId=1] Apr 13 12:26:29 ntop2 ntop[7446]: [MSGID9333267] **WARNING** Error: wrong bucketIdx /3.148 (expected=1750/real=0) [deviceId=1] -- Apr 13 12:31:23 ntop2 ntop[7484]: [MSGID9333267] **WARNING** Error: bad magic number (expected=1968/real=1750) [deviceId=1] Apr 13 12:31:23 ntop2 ntop[7484]: [MSGID9333267] **WARNING** Error: wrong bucketIdx /3.148 (expected=1750/real=0) [deviceId=1] -- Apr 13 12:36:17 ntop2 ntop[7519]: [MSGID9333267] **WARNING** Error: bad magic number (expected=1968/real=1307) [deviceId=1] Apr 13 12:36:17 ntop2 ntop[7519]: [MSGID9333267] **WARNING** Error: wrong bucketIdx /9.167 (expected=1307/real=0) [deviceId=1] -- Apr 13 12:46:05 ntop2 ntop[7592]: [MSGID0701710] **WARNING** Error: bad magic number (expected=1968/real=14506) Apr 13 12:46:05 ntop2 kernel: device eth0 left promiscuous mode -- Apr 15 11:47:35 ntop2 ntop[15471]: [MSGID9123622] **WARNING** Error: bad magic number [expected=1968/real=6422][deviceId=1] Apr 15 11:47:35 ntop2 kernel: device eth0 left promiscuous mode -- Apr 15 12:03:29 ntop2 ntop[15509]: [MSGID0701710] **WARNING** Error: bad magic number (expected=1968/real=15785) Apr 15 12:03:29 ntop2 kernel: device eth0 left promiscuous mode -- Apr 19 22:47:06 ntop2 ntop[15570]: [MSGID9333267] **WARNING** Error: bad magic number (expected=1968/real=1406) [deviceId=0] Apr 19 22:47:06 ntop2 ntop[15570]: [MSGID9333267] **WARNING** Error: wrong bucketIdx :2C:CD:D4/128.61 (expected=1406/real=0) [deviceId=0] -- Apr 19 22:50:10 ntop2 ntop[16054]: [MSGID9333267] **WARNING** Error: bad magic number (expected=1968/real=4449) [deviceId=1] Apr 19 22:50:10 ntop2 ntop[16054]: [MSGID9333267] **WARNING** Error: wrong bucketIdx /1.221 (expected=4449/real=0) [deviceId=1] -- Apr 19 23:00:37 ntop2 ntop[16127]: [MSGID9333267] **WARNING** Error: bad magic number (expected=1968/real=1489) [deviceId=0] Apr 19 23:00:37 ntop2 ntop[16127]: [MSGID8712897] **WARNING** Template 3254 has wrong size [actual=59099/expected=37214]: skipped -- Apr 19 23:05:34 ntop2 ntop[16166]: [MSGID9333267] **WARNING** Error: bad magic number (expected=1968/real=16162) [deviceId=1] Apr 19 23:05:34 ntop2 kernel: device eth0 left promiscuous mode -- Apr 19 23:20:57 ntop2 ntop[16277]: [MSGID9123622] **WARNING** Error: bad magic number [expected=1968/real=6666][deviceId=1] Apr 19 23:20:57 ntop2 kernel: device eth0 left promiscuous mode -- Apr 19 23:25:15 ntop2 ntop[16312]: [MSGID9123622] **WARNING** Error: bad magic number [expected=1968/real=11373][deviceId=1] Apr 19 23:25:15 ntop2 kernel: device eth0 left promiscuous mode -- Apr 19 23:30:47 ntop2 ntop[16350]: [MSGID9333267] **WARNING** Error: bad magic number (expected=1968/real=1406) [deviceId=0] Apr 19 23:30:47 ntop2 ntop[16350]: [MSGID9333267] **WARNING** Error: wrong bucketIdx :2C:CD:D4/128.61 (expected=1406/real=0) [deviceId=0] -- Apr 21 09:16:00 ntop2 ntop[22924]: [MSGID9333267] **WARNING** Error: bad magic number (expected=1968/real=6086) [deviceId=1] Apr 21 09:16:00 ntop2 ntop[22924]: [MSGID9333267] **WARNING** Error: wrong bucketIdx /82.132 (expected=6086/real=0) [deviceId=1] -- Apr 21 09:34:17 ntop2 ntop[23122]: [MSGID9333267] **WARNING** Error: bad magic number (expected=1968/real=1489) [deviceId=0] Apr 21 09:34:17 ntop2 ntop[23122]: [MSGID9333267] **WARNING** Error: wrong bucketIdx :2C:77:06/128.146 (expected=1489/real=0) [deviceId=0] -- Apr 21 13:01:23 ntop2 ntop[23887]: [MSGID9333267] **WARNING** Error: bad magic number (expected=1968/real=1406) [deviceId=0] Apr 21 13:01:23 ntop2 ntop[23887]: [MSGID9333267] **WARNING** Error: wrong bucketIdx :2C:CD:D4/128.61 (expected=1406/real=0) [deviceId=0] -- Apr 21 13:11:14 ntop2 ntop[23965]: [MSGID9123622] **WARNING** Error: bad magic number [expected=1968/real=9036][deviceId=1] Apr 21 13:11:14 ntop2 kernel: device eth0 left promiscuous mode -- Apr 21 13:15:31 ntop2 ntop[24000]: [MSGID9333267] **WARNING** Error: bad magic number (expected=1968/real=10372) [deviceId=1] Apr 21 13:15:31 ntop2 ntop[24000]: [MSGID9333267] **WARNING** Error: wrong bucketIdx /4.56 (expected=10372/real=0) [deviceId=1] -- Apr 21 13:35:44 ntop2 ntop[24148]: [MSGID9333267] **WARNING** Error: bad magic number (expected=1968/real=12815) [deviceId=1] Apr 21 13:35:44 ntop2 ntop[24148]: [MSGID9333267] **WARNING** Error: wrong bucketIdx /44.178 (expected=12815/real=0) [deviceId=1] -- Apr 21 13:39:20 ntop2 ntop[24218]: [MSGID9123622] **WARNING** Error: bad magic number [expected=1968/real=4514][deviceId=1] Apr 21 13:39:20 ntop2 kernel: device eth0 left promiscuous mode -- Apr 21 13:44:52 ntop2 ntop[24332]: [MSGID9123622] **WARNING** Error: bad magic number [expected=1968/real=16162][deviceId=1] Apr 21 13:44:52 ntop2 kernel: device eth0 left promiscuous mode -- Apr 21 14:06:03 ntop2 ntop[24458]: [MSGID9333267] **WARNING** Error: bad magic number (expected=1968/real=1406) [deviceId=0] Apr 21 14:06:03 ntop2 ntop[24458]: [MSGID9333267] **WARNING** Error: wrong bucketIdx :2C:CD:D4/128.61 (expected=1406/real=0) [deviceId=0] -- Apr 21 14:17:42 ntop2 ntop[24531]: [MSGID9333267] **WARNING** Error: bad magic number (expected=1968/real=1406) [deviceId=0] Apr 21 14:17:42 ntop2 ntop[24531]: [MSGID9333267] **WARNING** Error: wrong bucketIdx :2C:CD:D4/128.61 (expected=1406/real=0) [deviceId=0] -- Apr 21 14:53:44 ntop2 ntop[24629]: [MSGID9333267] **WARNING** Error: bad magic number (expected=1968/real=12610) [deviceId=0] Apr 21 14:53:44 ntop2 ntop[24629]: [MSGID9333267] **WARNING** Error: wrong bucketIdx :FF:44:11/180.1 (expected=12610/real=0) [deviceId=0] -- Apr 21 15:04:44 ntop2 ntop[24684]: [MSGID9333267] **WARNING** Error: bad magic number (expected=1968/real=1406) [deviceId=0] Apr 21 15:04:44 ntop2 ntop[24684]: [MSGID9333267] **WARNING** Error: wrong bucketIdx :2C:CD:D4/128.61 (expected=1406/real=0) [deviceId=0] -- Apr 23 12:36:27 ntop2 ntop[15635]: [MSGID9333267] **WARNING** Error: bad magic number (expected=1968/real=1406) [deviceId=0] Apr 23 12:36:27 ntop2 ntop[15635]: [MSGID9333267] **WARNING** Error: wrong bucketIdx :2C:CD:D4/128.61 (expected=1406/real=0) [deviceId=0] -- Apr 23 12:37:43 ntop2 ntop[15678]: [MSGID9333267] **WARNING** Error: bad magic number (expected=1968/real=15043) [deviceId=1] Apr 23 12:37:43 ntop2 ntop[15678]: [MSGID9333267] **WARNING** Error: wrong bucketIdx /4.127 (expected=15043/real=0) [deviceId=1] -- Apr 23 12:48:06 ntop2 ntop[18017]: [MSGID9333267] **WARNING** Error: bad magic number (expected=1968/real=1406) [deviceId=0] Apr 23 12:48:06 ntop2 ntop[18017]: [MSGID9333267] **WARNING** Error: wrong bucketIdx :2C:CD:D4/128.61 (expected=1406/real=0) [deviceId=0] -- Apr 23 13:47:30 ntop2 ntop[20489]: [MSGID9333267] **WARNING** Error: bad magic number (expected=1968/real=1406) [deviceId=0] Apr 23 13:47:30 ntop2 ntop[20489]: [MSGID9333267] **WARNING** Error: wrong bucketIdx :2C:CD:D4/128.61 (expected=1406/real=0) [deviceId=0] -- Apr 23 14:04:10 ntop2 ntop[2083]: [MSGID9333267] **WARNING** Error: bad magic number (expected=1968/real=1406) [deviceId=0] Apr 23 14:04:10 ntop2 ntop[2083]: [MSGID9333267] **WARNING** Error: wrong bucketIdx :2C:CD:D4/128.61 (expected=1406/real=0) [deviceId=0] -- Apr 23 21:25:39 ntop2 ntop[16505]: [MSGID9333267] **WARNING** Error: bad magic number (expected=1968/real=1489) [deviceId=0] Apr 23 21:25:39 ntop2 ntop[16505]: [MSGID9333267] **WARNING** Error: wrong bucketIdx :2C:77:06/128.146 (expected=1489/real=0) [deviceId=0] -- Apr 23 21:33:46 ntop2 ntop[16505]: [MSGID9123622] **WARNING** Error: bad magic number [expected=1968/real=1212][deviceId=0] Apr 23 21:40:01 ntop2 crond(pam_unix)[16603]: session opened for user root by (uid=0) -- Apr 24 16:58:39 ntop2 ntop[2252]: [MSGID9333267] **WARNING** Error: bad magic number (expected=1968/real=14839) [deviceId=1] Apr 24 16:58:39 ntop2 ntop[2252]: [MSGID9333267] **WARNING** Error: wrong bucketIdx /55.74 (expected=14839/real=0) [deviceId=1] -- Apr 24 16:59:54 ntop2 ntop[2282]: [MSGID9333267] **WARNING** Error: bad magic number (expected=1968/real=4399) [deviceId=1] Apr 24 16:59:54 ntop2 ntop[2282]: [MSGID9333267] **WARNING** Error: wrong bucketIdx /1.147 (expected=4399/real=0) [deviceId=1] -- Apr 25 10:08:55 ntop2 ntop[2365]: [MSGID9333267] **WARNING** Error: bad magic number (expected=1968/real=9046) [deviceId=1] Apr 25 10:08:55 ntop2 ntop[2365]: [MSGID9333267] **WARNING** Error: wrong bucketIdx /.234 (expected=9046/real=0) [deviceId=1] -- Apr 25 11:00:25 ntop2 ntop[3807]: [MSGID9333267] **WARNING** Error: bad magic number (expected=1968/real=9046) [deviceId=1] Apr 25 11:00:25 ntop2 ntop[3807]: [MSGID9333267] **WARNING** Error: wrong bucketIdx /.234 (expected=9046/real=0) [deviceId=1] ---------- Forwarded message ---------- Date: Sat, 23 Apr 2005 13:56:06 +0200 From: Robbert Kouprie <[EMAIL PROTECTED]> Reply-To: [email protected] To: [email protected] Subject: [Ntop-dev] Ntop segfaulting Hi, Suddenly I'm experiencing segfaults. This is ntop CVS on Redhat Enterprise 4. I am using Netflow. For some reason I expect a change in network configuration, but still ntop shouldn't segfault. I collected 2 backtraces: one from ntop CVS from April 4th, and one from CVS current as of today. Hope you can see what's wrong. Regards, Robbert cvs from April 4th: Program received signal SIGSEGV, Segmentation fault. [Switching to Thread -1300034640 (LWP 19298)] 0x003e5cd6 in malloc_consolidate () from /lib/tls/libc.so.6 (gdb) bt #0 0x003e5cd6 in malloc_consolidate () from /lib/tls/libc.so.6 #1 0x003e6dd3 in _int_malloc () from /lib/tls/libc.so.6 #2 0x003e8ba1 in malloc () from /lib/tls/libc.so.6 #3 0x003deaf3 in open_memstream () from /lib/tls/libc.so.6 #4 0x0044795e in vsyslog () from /lib/tls/libc.so.6 #5 0x0044800f in syslog () from /lib/tls/libc.so.6 #6 0x0101755d in traceEvent (eventTraceLevel=2, file=0x10247b5 "hash.c", line=836, format=0x1024a58 "Error: bad magic number (expected=%d/real=%d) [deviceId=%d]") at util.c:2665 #7 0x00ff576b in lookupHost (hostIpAddress=0xb282de00, ether_addr=0xb282e2c0 "", vlanId=-1, checkForMultihoming=1 '\001', forceUsingIPaddress=0 '\0', actualDeviceId=0) at hash.c:836 #8 0x01000ae8 in processIpPkt (bp=0xb282e31e "E", h=0xb2830390, length=1460, ether_src=0xb282e2c6 "", ether_dst=0xb282e2c0 "", actualDeviceId=0, vlanId=-1) at pbuf.c:1086 #9 0x010065e0 in processPacket (_deviceId=0x0, h=0xb2830390, p=0xb282e310 "") at pbuf.c:3480 #10 0x010092e1 in queuePacket (_deviceId=0x0, h=0xb2830390, p=0x8a0acc2 "") at pbuf.c:2287 #11 0x006854ae in ?? () from /usr/lib/libpcap.so.0.8.3 #12 0x00000000 in ?? () CVS current: Program received signal SIGSEGV, Segmentation fault. [Switching to Thread -1286157392 (LWP 20499)] 0x003926ef in lookupHost (hostIpAddress=0xb3569de0, ether_addr=0xb356a2c0 "", vlanId=-1, checkForMultihoming=1 '\001', forceUsingIPaddress=0 '\0', actualDeviceId=0) at hash.c:835 835 if(el->magic != CONST_MAGIC_NUMBER) { (gdb) bt #0 0x003926ef in lookupHost (hostIpAddress=0xb3569de0, ether_addr=0xb356a2c0 "", vlanId=-1, checkForMultihoming=1 '\001', forceUsingIPaddress=0 '\0', actualDeviceId=0) at hash.c:835 #1 0x0039daa0 in processIpPkt (bp=0xb356a31e "E", h=0xb356c390, length=1448, ether_src=0xb356a2c6 "", ether_dst=0xb356a2c0 "", actualDeviceId=0, vlanId=-1) at pbuf.c:1086 #2 0x003a36b0 in processPacket (_deviceId=0x0, h=0xb356c390, p=0xb356a310 "") at pbuf.c:3525 #3 0x003a6429 in queuePacket (_deviceId=0x0, h=0xb356c390, p=0x948ad22 "") at pbuf.c:2300 #4 0x002634ae in ?? () from /usr/lib/libpcap.so.0.8.3 #5 0x00000000 in ?? () (gdb) _______________________________________________ Ntop-dev mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop-dev _______________________________________________ Ntop-dev mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop-dev _______________________________________________ Ntop-dev mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop-dev _______________________________________________ Ntop-dev mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop-dev
