I've got a couple of (probably related) problems with PF_RING 4.7.2
1) With PF_RING-enabled libpcap, Suricata would crash and ARGUS
complains of out-of-order timestamps every couple of hours. E.g.
Suricata core backtrace:
> Core was generated by `/opt/RDGsuricata/bin/suricata -i eth1 -c
> /etc/suricata/suricata.yaml -D'.
> Program terminated with signal 11, Segmentation fault.
> #0 0x00000000004d6af7 in snprintf (ts=0x27e7d00,
> str=0x7f53277fdd70 "08/03/2011-11:41:26.410990",
> size=<value optimized out>) at /usr/include/bits/stdio2.h:65
> 65 return __builtin___snprintf_chk (__s, __n, __USE_FORTIFY_LEVEL - 1,
> #0 0x00000000004d6af7 in snprintf (ts=0x27e7d00,
> str=0x7f53277fdd70 "08/03/2011-11:41:26.410990",
> size=<value optimized out>) at /usr/include/bits/stdio2.h:65
> No locals.
> #1 CreateTimeString (ts=0x27e7d00,
> str=0x7f53277fdd70 "08/03/2011-11:41:26.410990",
> size=<value optimized out>) at log-httplog.c:103
> time = -6564470811307293241
> local_tm = {tm_sec = 44, tm_min = 4, tm_hour = 6, tm_mday = 3,
> tm_mon = 775172913, tm_year = -1861368698, tm_wday = 4,
> tm_yday = 3420209, tm_isdst = 0, tm_gmtoff = -75,
> tm_zone = 0x1988790 "LMT"}
> t = <value optimized out>
> #2 0x00000000004d6c70 in LogHttpLogIPv4 (tv=<value optimized out>,
> p=0x27e7cc0, data=0x32a095d0, pq=<value optimized out>,
> postpq=<value optimized out>) at log-httplog.c:149
> timebuf = "08/03/2011-11:41:26.410990\000
> S\177\000\000\000\000\000\000\000\000\000\000\016", '\000' <repeats 15
> times>, " \000\000\034S\177\000"
> idx = <value optimized out>
> proto = <value optimized out>
> r = 0
> logged = <value optimized out>
> loggable = <value optimized out>
> htp_state = 0x0
> tx = <value optimized out>
> srcip = "192.171.\000q1z58\361D"
> dstip = "134.157.176.104"
> sp = <value optimized out>
> dp = 80
> #3 0x00000000004c9069 in TmThreadsSlotVarRun (td=0x182b4db0)
> at tm-threads.c:425
> r = <value optimized out>
> s = 0x182b7790
> #4 TmThreadsSlotVar (td=0x182b4db0) at tm-threads.c:517
> s = 0x182b4e90
> p = 0x27e7cc0
> r = <value optimized out>
> slot = 0x0
> #5 0x00007f532f95c9ca in start_thread () from /lib/libpthread.so.0
> No symbol table info available.
> #6 0x00007f532f26b70d in clone () from /lib/libc.so.6
> No symbol table info available.
> #7 0x0000000000000000 in ?? ()
> No symbol table info available.
However, Suricata with native PF_RING is fine.
ARGUS (in daemon.log) is showing (I've got a check script restarting it
automatically if it dies):-
> Aug 25 20:50:02 vinms1 argus[9140]: 25 Aug 11 20:50:02.416348
> ArgusGetInterfaceStatus: interface eth1 is up
> Aug 25 20:52:37 vinms1 argus[9140]: 25 Aug 11 20:52:37.850281 ArgusInterface
> timestamps wayyy out of order: now 1314301957 then -1476065857
> Aug 25 20:52:37 vinms1 argus[9140]: 25 Aug 11 20:52:37.850517 ArgusInterface
> timestamps wayyy out of order: now 67108864 then 1314301957
> Aug 25 20:52:42 vinms1 argus[9140]: 25 Aug 11 20:52:42.850627
> ArgusGenerateRecord: packet size type not defined
> Aug 25 21:04:03 vinms1 argus[9587]: 25 Aug 11 21:04:03.714063 started
> Aug 25 21:04:03 vinms1 argus[9587]: 25 Aug 11 21:04:03.715963 started
> Aug 25 21:04:03 vinms1 argus[9587]: 25 Aug 11 21:04:03.864859
> ArgusGetInterfaceStatus: interface eth1 is up
> Aug 25 21:18:59 vinms1 argus[9587]: 25 Aug 11 21:18:59.352774 ArgusInterface
> timestamps wayyy out of order: now -2133983232 then 224
> Aug 25 21:19:04 vinms1 argus[9587]: 25 Aug 11 21:19:04.353456
> ArgusGenerateRecord: packet size type not defined
> Aug 25 21:20:02 vinms1 argus[10164]: 25 Aug 11 21:20:02.035960 started
> Aug 25 21:20:02 vinms1 argus[10164]: 25 Aug 11 21:20:02.037841 started
> Aug 25 21:20:02 vinms1 argus[10164]: 25 Aug 11 21:20:02.126142
> ArgusGetInterfaceStatus: interface eth1 is up
> Aug 26 02:26:57 vinms1 argus[10164]: 26 Aug 11 02:26:57.924060 ArgusInterface
> timestamps wayyy out of order: now 1314321751 then -1962934272
> Aug 26 03:04:04 vinms1 argus[20982]: 26 Aug 11 03:04:04.097349 started
> Aug 26 03:04:04 vinms1 argus[20982]: 26 Aug 11 03:04:04.099271 started
> Aug 26 03:04:04 vinms1 argus[20982]: 26 Aug 11 03:04:04.156049
> ArgusGetInterfaceStatus: interface eth1 is up
> Aug 26 09:51:31 vinms1 argus[20982]: 26 Aug 11 09:51:31.340779 ArgusInterface
> timestamps wayyy out of order: now 287696 then 1314348691
> Aug 26 09:51:31 vinms1 argus[20982]: 26 Aug 11 09:51:31.341377 ArgusInterface
> timestamps wayyy out of order: now 66 then 1314348691
> Aug 26 09:51:36 vinms1 argus[20982]: 26 Aug 11 09:51:36.341697
> ArgusGenerateRecord: packet size type not defined
2) The kernel is panicking every few days with pf_ring errors. Alas
nothing in the logs, but I've got screen dumps of some of the console
output (using Dell DRAC cards).
E.g. (copying by hand!) call trace:
? skb_ring_handler+0x92d/0xf10 [pf_ring]
? read_tsc+0x9/0x20
? ktime_get+0x63/0xe0
? lapic_next_event+0x1d/0x30
? clockevents_program_event+0x54/0xa0
? tick_dev_program_event+0x68/0xd0
? __slab_alloc+0x92/0x2d0
? swiotlb_dma_mapping_error+0x18/0x30
? e1000_receive_skb+0xb8/0xf0 [e1000e]
And
? __tcp_ack_snd_check+0x5e/0xa0
...
? swiotlb_dma_mapping_error+0x18/0x30
? e1000_receive_skb+0xb8/0xf0 [e1000e]
I'm wondering whether some of the usec code is causing problems with the
e1000e driver? PF_RING 4.6.5 was OK, but 4.7.1 wasn't, I think.
Best Wishes,
Chris
--
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
Christopher Wakelin, [email protected]
IT Services Centre, The University of Reading, Tel: +44 (0)118 378 2908
Whiteknights, Reading, RG6 6AF, UK Fax: +44 (0)118 975 3094
_______________________________________________
Ntop-misc mailing list
[email protected]
http://listgateway.unipi.it/mailman/listinfo/ntop-misc
