-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi Adrian,
- --On May 12, 2008 10:36:10 +0300 Adrian Popa <[EMAIL PROTECTED]> wrote:
| I hate mondays.... Mainly because it's the day when I find nfsen crashed...
| :)
This is really very unfortunate to hear. The weekend phenomena: NfSen knows no
weekends. Every day is equal. ( So far no unions
coplained :)
What I saw in the logs is, that NfSen stuck after the module init phase - or
you cut out all other NfSen logs. If it really
stuck. So the sequence:
syslog("info", "Comm server started: [$$]");
Nfcomm::LoadPlugins();
semsignal($semlock);
Nfcomm::RunServer($server);
Nfcomm::CleanupPlugins();
is somewhat stuck.
I guess the semsignal did not work correctly here and blocked nfsen forever.
Unfortunately I will be away for the next 2 weeks and need more time to look
into this issue.
I'll be back.
- Peter
snip log entries ...
| May 11 04:06:47 localhost prefixStats: plugin 'PortTracker': Profile plugin:
| 1, Alert condition plugin: 0, Alert action plugin: 0
| May 11 04:06:47 localhost prefixStats: Plugins for profile : ./live
| - floodsearch,prefixStats,PortTracker
| May 11 04:06:47 localhost prefixStats: No site specific lookup module found
| May 11 04:06:47 localhost snmpd: snmpd shutdown succeeded
| May 11 04:06:48 localhost snmpd: snmpd startup succeeded
|
| After the log rotation, I get mostly the same output in the logs, but I *no
| longer get 'expire' messages!*
| This is the log just before the disk fills up and just after it is manually
| resurected:
|
|
| May 11 18:50:04 localhost /usr/local/bin/nfcapd[16121]: Ident: '7613ban'
| Flows: 7566, Packets: 17085, Bytes: 18435994, Sequence Errors: 0, Bad
| Packets: 0
| May 11 18:50:05 localhost /usr/local/bin/nfcapd[16124]: Ident: '7613cnt2'
| Flows: 79851, Packets: 342118, Bytes: 361389750, Sequence Errors: 0, Bad
| Packets: 0
| May 11 18:50:10 localhost /usr/local/bin/nfcapd[16154]: Ident: '7606_lab_1'
| Flows: 0, Packets: 0, Bytes: 0, Sequence Errors: 0, Bad Packets: 0
| May 11 18:50:10 localhost /usr/local/bin/nfcapd[16142]: Ident: '7304bcnt2'
| Flows: 0, Packets: 0, Bytes: 0, Sequence Errors: 0, Bad Packets: 0
| May 11 18:50:10 localhost /usr/local/bin/nfcapd[16157]: Ident: '7606sv1'
| Flows: 0, Packets: 0, Bytes: 0, Sequence Errors: 0, Bad Packets: 0
| May 11 18:55:00 localhost /usr/local/bin/nfcapd[16133]: Ident:
| '7613_Sud_Est' Flows: 66047, Packets: 114314, Bytes: 91445272, Sequence
| Errors: 0, Bad Packets
| : 0
| May 11 18:55:00 localhost /usr/local/bin/nfcapd[16151]: Ident:
| '12410bra2igw' Flows: 1205523, Packets: 3980633, Bytes: 2218349278, Sequence
| Errors: 0, Bad Pa
| ckets: 0
| May 11 18:55:00 localhost /usr/local/bin/nfcapd[16148]: Ident: '7304bb2'
| Flows: 2781150, Packets: 33537442, Bytes: 20163089383, Sequence Errors:
| 92704, Bad P
| ackets: 0
| May 11 18:55:00 localhost /usr/local/bin/nfcapd[16136]: Ident: '12410bcnt2'
| Flows: 1646763, Packets: 10442317, Bytes: 7425684627, Sequence Errors: 0,
| Bad Pac
| kets: 0
| May 11 18:55:00 localhost /usr/local/bin/nfcapd[16139]: Ident:
| '12410intx1IGW' Flows: 968243, Packets: 3123603, Bytes: 2066919250, Sequence
| Errors: 4, Bad Pa
| ckets: 0
| May 11 18:55:00 localhost /usr/local/bin/nfcapd[16145]: Ident:
| '12410FFanc1IGW' Flows: 978003, Packets: 2893129, Bytes: 1727351983,
| Sequence Errors: 0, Bad P
| ackets: 0
| May 11 18:55:00 localhost /usr/local/bin/nfcapd[16127]: Ident:
| '6509Victoria' Flows: 182548, Packets: 717734, Bytes: 682151286, Sequence
| Errors: 2, Bad Packe
| ts: 0
| May 11 18:55:00 localhost /usr/local/bin/nfcapd[16124]: Ident: '7613cnt2'
| Flows: 75863, Packets: 318415, Bytes: 335156764, Sequence Errors: 0, Bad
| Packets: 0
| May 11 18:55:01 localhost /usr/local/bin/nfcapd[16121]: Ident: '7613ban'
| Flows: 7632, Packets: 17623, Bytes: 18922983, Sequence Errors: 0, Bad
| Packets: 0
| May 11 18:55:01 localhost crond(pam_unix)[8885]: session opened for user
| root by (uid=0)
| May 11 18:55:01 localhost crond(pam_unix)[8886]: session opened for user
| root by (uid=0)
| May 11 18:55:01 localhost crond(pam_unix)[8888]: session opened for user
| root by (uid=0)
| May 11 18:55:02 localhost crond(pam_unix)[8886]: session closed for user
| root
| May 11 18:55:02 localhost crond(pam_unix)[8885]: session closed for user
| root
| May 11 18:55:10 localhost /usr/local/bin/nfcapd[16154]: Ident: '7606_lab_1'
| Flows: 0, Packets: 0, Bytes: 0, Sequence Errors: 0, Bad Packets: 0
| May 11 18:55:10 localhost /usr/local/bin/nfcapd[16142]: Ident: '7304bcnt2'
| Flows: 0, Packets: 0, Bytes: 0, Sequence Errors: 0, Bad Packets: 0
| May 11 18:55:10 localhost /usr/local/bin/nfcapd[16157]: Ident: '7606sv1'
| Flows: 0, Packets: 0, Bytes: 0, Sequence Errors: 0, Bad Packets: 0
| May 11 18:57:01 localhost crond(pam_unix)[8897]: session opened for user
| root by (uid=0)
| May 11 18:57:02 localhost crond(pam_unix)[8888]: session closed for user
| root
| *May 11 18:59:21 localhost /usr/local/bin/nfcapd[16148]: Failed to write
| output buffer to disk: 'No space left on device'
| May 11 18:59:21 localhost last message repeated 134 times
| May 11 18:59:21 localhost /usr/local/bin/nfcapd[16148]: Process_v9: output
| buffer size error. Abort v9 record processing
| May 11 18:59:21 localhost /usr/local/bin/nfcapd[16148]: Failed to write
| output buffer to disk: 'No space left on device'*
| May 11 18:59:21 localhost /usr/local/bin/nfcapd[16148]: Process_v9: output
| buffer size error. Abort v9 record processing
| May 11 18:59:21 localhost /usr/local/bin/nfcapd[16148]: Failed to write
| output buffer to disk: 'No space left on device'
| May 11 18:59:21 localhost /usr/local/bin/nfcapd[16148]: Process_v9: output
| buffer size error. Abort v9 record processing
| May 11 18:59:21 localhost /usr/local/bin/nfcapd[16148]: Failed to write
| output buffer to disk: 'No space left on device'
| May 11 18:59:21 localhost /usr/local/bin/nfcapd[16148]: Process_v9: output
| buffer size error. Abort v9 record processing
|
| ...
|
| May 12 08:34:54 localhost /usr/local/bin/nfcapd[16133]: Process_v9: output
| buffer size error. Abort v9 record processing
| May 12 08:34:54 localhost /usr/local/bin/nfcapd[16133]: Failed to write
| output buffer to disk: 'Bad file descriptor'
| May 12 08:34:54 localhost /usr/local/bin/nfcapd[16133]: Process_v9: output
| buffer size error. Abort v9 record processing
| May 12 08:34:54 localhost /usr/local/bin/nfcapd[16133]: Failed to write
| output buffer to disk: 'Bad file descriptor'
| May 12 08:34:54 localhost /usr/local/bin/nfcapd[16133]: Process_v9: output
| buffer size error. Abort v9 record processing
| May 12 08:34:54 localhost /usr/local/bin/nfcapd[16133]: Failed to write
| output buffer to disk: 'Bad file descriptor'
| May 12 08:35:00 localhost /usr/local/bin/nfcapd[16148]: Ident: '7304bb2'
| Flows: 756814, Packets: 5872977, Bytes: 3913082766, Sequence Errors: 46564,
| Bad Pack
| ets: 0
| May 12 08:35:00 localhost /usr/local/bin/nfcapd[16151]: Ident:
| '12410bra2igw' Flows: 181594, Packets: 557380, Bytes: 322579617, Sequence
| Errors: 0, Bad Packe
| ts: 0
| May 12 08:35:00 localhost /usr/local/bin/nfcapd[16139]: Ident:
| '12410intx1IGW' Flows: 190967, Packets: 413212, Bytes: 305997877, Sequence
| Errors: 16, Bad Pac
| kets: 0
| May 12 08:35:00 localhost /usr/local/bin/nfcapd[16136]: Ident: '12410bcnt2'
| Flows: 297979, Packets: 1079903, Bytes: 754980175, Sequence Errors: 187, Bad
| Pack
| ets: 0
| May 12 08:35:00 localhost /usr/local/bin/nfcapd[16145]: Ident:
| '12410FFanc1IGW' Flows: 164247, Packets: 361482, Bytes: 226356155, Sequence
| Errors: 0, Bad Pac
| kets: 0
| May 12 08:35:00 localhost /usr/local/bin/nfcapd[16133]: Failed to write
| output buffer to disk: 'Bad file descriptor'
| May 12 08:35:00 localhost /usr/local/bin/nfcapd[16133]: lseek failed: 'Bad
| file descriptor'
| May 12 08:35:00 localhost /usr/local/bin/nfcapd[16133]: Ident:
| '7613_Sud_Est' Flows: 59152, Packets: 227308, Bytes: 3724884305, Sequence
| Errors: 2, Bad Packe
| ts: 0
| May 12 08:35:01 localhost crond(pam_unix)[19182]: session opened for user
| root by (uid=0)
| May 12 08:35:01 localhost crond(pam_unix)[19181]: session opened for user
| root by (uid=0)
| May 12 08:35:01 localhost crond(pam_unix)[19183]: session opened for user
| root by (uid=0)
| May 12 08:35:01 localhost crond(pam_unix)[19182]: session closed for user
| root
| May 12 08:35:02 localhost crond(pam_unix)[19181]: session closed for user
| root
| May 12 08:35:04 localhost /usr/local/bin/nfcapd[16121]: Ident: '7613ban'
| Flows: 3695, Packets: 9990, Bytes: 10789322, Sequence Errors: 0, Bad
| Packets: 0
| May 12 08:35:05 localhost /usr/local/bin/nfcapd[16124]: Ident: '7613cnt2'
| Flows: 30626, Packets: 120622, Bytes: 133965438, Sequence Errors: 0, Bad
| Packets: 0
| May 12 08:35:07 localhost /usr/local/bin/nfcapd[16127]: Ident:
| '6509Victoria' Flows: 44264, Packets: 179778, Bytes: 180333303, Sequence
| Errors: 0, Bad Packet
| s: 0
| May 12 08:35:10 localhost /usr/local/bin/nfcapd[16142]: Ident: '7304bcnt2'
| Flows: 0, Packets: 0, Bytes: 0, Sequence Errors: 0, Bad Packets: 0
| May 12 08:35:10 localhost /usr/local/bin/nfcapd[16157]: Ident: '7606sv1'
| Flows: 0, Packets: 0, Bytes: 0, Sequence Errors: 0, Bad Packets: 0
| May 12 08:35:10 localhost /usr/local/bin/nfcapd[16154]: Ident: '7606_lab_1'
| Flows: 0, Packets: 0, Bytes: 0, Sequence Errors: 0, Bad Packets: 0
| May 12 08:35:13 localhost kernel: Kernel logging (proc) stopped.
| May 12 08:35:13 localhost kernel: Kernel log daemon terminating.
| May 12 08:35:14 localhost syslog: klogd shutdown succeeded
| May 12 08:35:14 localhost exiting on signal 15
| May 12 08:35:14 localhost syslogd 1.4.1: restart.
| May 12 08:35:14 localhost syslog: syslogd startup succeeded
| May 12 08:35:14 localhost kernel: klogd 1.4.1, log source = /proc/kmsg
| started.
| May 12 08:35:14 localhost syslog: klogd startup succeeded
| May 12 08:35:15 localhost /usr/local/bin/nfcapd[16121]: Ident: '7613ban'
| Flows: 201, Packets: 504, Bytes: 566724, Sequence Errors: 0, Bad Packets: 0
| May 12 08:35:15 localhost /usr/local/bin/nfcapd[16121]: Updating statinfo in
| directory '/data/nfsen/profiles/live/7613ban'
| May 12 08:35:15 localhost /usr/local/bin/nfcapd[16121]: Terminating nfcapd.
| May 12 08:35:16 localhost /usr/local/bin/nfcapd[16124]: Ident: '7613cnt2'
| Flows: 1966, Packets: 7717, Bytes: 8427333, Sequence Errors: 0, Bad Packets:
| 0
| May 12 08:35:16 localhost /usr/local/bin/nfcapd[16124]: Updating statinfo in
| directory '/data/nfsen/profiles/live/7613cnt2'
| May 12 08:35:16 localhost /usr/local/bin/nfcapd[16124]: Terminating nfcapd.
| May 12 08:35:14 localhost syslog: syslogd shutdown succeeded
| May 12 08:35:17 localhost /usr/local/bin/nfcapd[16127]: Ident:
| '6509Victoria' Flows: 3859, Packets: 15919, Bytes: 16361833, Sequence
| Errors: 0, Bad Packets:
| 0
| May 12 08:35:17 localhost /usr/local/bin/nfcapd[16127]: Updating statinfo in
| directory '/data/nfsen/profiles/live/6509Victoria'
| May 12 08:35:17 localhost /usr/local/bin/nfcapd[16127]: Terminating nfcapd.
| May 12 08:35:19 localhost /usr/local/bin/nfcapd[16133]: Ident:
| '7613_Sud_Est' Flows: 1573, Packets: 2546, Bytes: 2207406, Sequence Errors:
| 0, Bad Packets: 0
| May 12 08:35:19 localhost /usr/local/bin/nfcapd[16133]: Updating statinfo in
| directory '/data/nfsen/profiles/live/7613_Sud_Est'
| May 12 08:35:19 localhost /usr/local/bin/nfcapd[16133]: Terminating nfcapd.
| May 12 08:35:20 localhost /usr/local/bin/nfcapd[16136]: Ident: '12410bcnt2'
| Flows: 56178, Packets: 223413, Bytes: 143377671, Sequence Errors: 0, Bad
| Packets:
| 0
| May 12 08:35:20 localhost /usr/local/bin/nfcapd[16136]: Updating statinfo in
| directory '/data/nfsen/profiles/live/12410bcnt2'
| May 12 08:35:20 localhost /usr/local/bin/nfcapd[16136]: Terminating nfcapd.
| May 12 08:35:21 localhost /usr/local/bin/nfcapd[16139]: Ident:
| '12410intx1IGW' Flows: 34088, Packets: 73665, Bytes: 52943257, Sequence
| Errors: 0, Bad Packets
| : 0
| May 12 08:35:21 localhost /usr/local/bin/nfcapd[16139]: Updating statinfo in
| directory '/data/nfsen/profiles/live/12410intx1IGW'
| May 12 08:35:21 localhost /usr/local/bin/nfcapd[16139]: Terminating nfcapd.
| May 12 08:35:22 localhost /usr/local/bin/nfcapd[16142]: Ident: '7304bcnt2'
| Flows: 0, Packets: 0, Bytes: 0, Sequence Errors: 0, Bad Packets: 0
| May 12 08:35:22 localhost /usr/local/bin/nfcapd[16142]: Updating statinfo in
| directory '/data/nfsen/profiles/live/7304bcnt2'
| May 12 08:35:22 localhost /usr/local/bin/nfcapd[16142]: Terminating nfcapd.
| May 12 08:35:23 localhost /usr/local/bin/nfcapd[16145]: Ident:
| '12410FFanc1IGW' Flows: 34117, Packets: 80188, Bytes: 49051491, Sequence
| Errors: 0, Bad Packet
| s: 0
| May 12 08:35:23 localhost /usr/local/bin/nfcapd[16145]: Updating statinfo in
| directory '/data/nfsen/profiles/live/12410FFanc1IGW'
| May 12 08:35:23 localhost /usr/local/bin/nfcapd[16145]: Terminating nfcapd.
| May 12 08:35:24 localhost /usr/local/bin/nfcapd[16148]: Ident: '7304bb2'
| Flows: 181800, Packets: 1059232, Bytes: 621547318, Sequence Errors: 6060,
| Bad Packet
| s: 0
| May 12 08:35:24 localhost /usr/local/bin/nfcapd[16148]: Updating statinfo in
| directory '/data/nfsen/profiles/live/7304bb2'
| May 12 08:35:24 localhost /usr/local/bin/nfcapd[16148]: Terminating nfcapd.
| May 12 08:35:25 localhost /usr/local/bin/nfcapd[16151]: Ident:
| '12410bra2igw' Flows: 41760, Packets: 129604, Bytes: 74492468, Sequence
| Errors: 0, Bad Packets
| : 0
| May 12 08:35:25 localhost /usr/local/bin/nfcapd[16151]: Updating statinfo in
| directory '/data/nfsen/profiles/live/12410bra2igw'
| May 12 08:35:25 localhost /usr/local/bin/nfcapd[16151]: Terminating nfcapd.
| May 12 08:35:26 localhost /usr/local/bin/nfcapd[16154]: Ident: '7606_lab_1'
| Flows: 0, Packets: 0, Bytes: 0, Sequence Errors: 0, Bad Packets: 0
| May 12 08:35:27 localhost /usr/local/bin/nfcapd[16154]: Updating statinfo in
| directory '/data/nfsen/profiles/live/7606_lab_1'
| May 12 08:35:27 localhost /usr/local/bin/nfcapd[16154]: Terminating nfcapd.
| May 12 08:35:28 localhost /usr/local/bin/nfcapd[16157]: Ident: '7606sv1'
| Flows: 0, Packets: 0, Bytes: 0, Sequence Errors: 0, Bad Packets: 0
| May 12 08:35:28 localhost /usr/local/bin/nfcapd[16157]: Updating statinfo in
| directory '/data/nfsen/profiles/live/7606sv1'
| May 12 08:35:28 localhost /usr/local/bin/nfcapd[16157]: Terminating nfcapd.
| May 12 08:35:31 localhost /usr/local/bin/nfcapd[21070]: Standard setsockopt,
| SO_RCVBUF is 110592 Requested length is 200000 bytes
| May 12 08:35:31 localhost /usr/local/bin/nfcapd[21070]: System set
| setsockopt, SO_RCVBUF to 262142 bytes
| May 12 08:35:31 localhost /usr/local/bin/nfcapd[21072]: Startup.
| May 12 08:35:31 localhost /usr/local/bin/nfcapd[21072]: Process_v9: New
| exporter domain 516
| May 12 08:35:31 localhost /usr/local/bin/nfcapd[21073]: Standard setsockopt,
| SO_RCVBUF is 110592 Requested length is 200000 bytes
| May 12 08:35:31 localhost /usr/local/bin/nfcapd[21073]: System set
| setsockopt, SO_RCVBUF to 262142 bytes
| May 12 08:35:31 localhost /usr/local/bin/nfcapd[21075]: Startup.
| May 12 08:35:31 localhost /usr/local/bin/nfcapd[21076]: Standard setsockopt,
| SO_RCVBUF is 110592 Requested length is 200000 bytes
| May 12 08:35:31 localhost /usr/local/bin/nfcapd[21070]: System set
| setsockopt, SO_RCVBUF to 262142 bytes
| May 12 08:35:31 localhost /usr/local/bin/nfcapd[21072]: Startup.
| May 12 08:35:31 localhost /usr/local/bin/nfcapd[21072]: Process_v9: New
| exporter domain 516
| May 12 08:35:31 localhost /usr/local/bin/nfcapd[21073]: Standard setsockopt,
| SO_RCVBUF is 110592 Requested length is 200000 bytes
| May 12 08:35:31 localhost /usr/local/bin/nfcapd[21073]: System set
| setsockopt, SO_RCVBUF to 262142 bytes
| May 12 08:35:31 localhost /usr/local/bin/nfcapd[21075]: Startup.
| May 12 08:35:31 localhost /usr/local/bin/nfcapd[21076]: Standard setsockopt,
| SO_RCVBUF is 110592 Requested length is 200000 bytes
| May 12 08:35:31 localhost /usr/local/bin/nfcapd[21076]: System set
| setsockopt, SO_RCVBUF to 262142 bytes
| May 12 08:35:31 localhost /usr/local/bin/nfcapd[21078]: Startup.
| May 12 08:35:31 localhost /usr/local/bin/nfcapd[21079]: Standard setsockopt,
| SO_RCVBUF is 110592 Requested length is 200000 bytes
| May 12 08:35:31 localhost /usr/local/bin/nfcapd[21079]: System set
| setsockopt, SO_RCVBUF to 262142 bytes
| May 12 08:35:31 localhost /usr/local/bin/nfcapd[21081]: Startup.
| May 12 08:35:31 localhost /usr/local/bin/nfcapd[21082]: Standard setsockopt,
| SO_RCVBUF is 110592 Requested length is 200000 bytes
| May 12 08:35:31 localhost /usr/local/bin/nfcapd[21082]: System set
| setsockopt, SO_RCVBUF to 262142 bytes
| May 12 08:35:31 localhost /usr/local/bin/nfcapd[21084]: Startup.
| May 12 08:35:32 localhost /usr/local/bin/nfcapd[21085]: Standard setsockopt,
| SO_RCVBUF is 110592 Requested length is 200000 bytes
| May 12 08:35:32 localhost /usr/local/bin/nfcapd[21085]: System set
| setsockopt, SO_RCVBUF to 262142 bytes
| May 12 08:35:32 localhost /usr/local/bin/nfcapd[21087]: Startup.
| May 12 08:35:32 localhost /usr/local/bin/nfcapd[21087]: Process_v9: New
| exporter domain 258
| May 12 08:35:32 localhost /usr/local/bin/nfcapd[21081]: Process_v9: New
| exporter domain 0
| May 12 08:35:32 localhost /usr/local/bin/nfcapd[21078]: Process_v9: New
| exporter domain 518
| May 12 08:35:32 localhost /usr/local/bin/nfcapd[21078]: Process_v9: [518]
| Add template 257
| May 12 08:35:32 localhost /usr/local/bin/nfcapd[21078]: Process_v9: [518]
| Add template 256
| May 12 08:35:32 localhost /usr/local/bin/nfcapd[21087]: Process_v9: New
| exporter domain 257
| May 12 08:35:32 localhost /usr/local/bin/nfcapd[21087]: Process_v9: [257]
| Add template 257
| May 12 08:35:32 localhost /usr/local/bin/nfcapd[21087]: Process_v9: New
| exporter domain 262
| May 12 08:35:32 localhost /usr/local/bin/nfcapd[21087]: Process_v9: [262]
| Add template 257
| May 12 08:35:32 localhost /usr/local/bin/nfcapd[21087]: Process_v9: New
| exporter domain 261
| May 12 08:35:32 localhost /usr/local/bin/nfcapd[21087]: Process_v9: [261]
| Add template 257
| May 12 08:35:33 localhost /usr/local/bin/nfcapd[21087]: Process_v9: [258]
| Add template 256
| May 12 08:35:33 localhost /usr/local/bin/nfcapd[21088]: Standard setsockopt,
| SO_RCVBUF is 110592 Requested length is 200000 bytes
| May 12 08:35:33 localhost /usr/local/bin/nfcapd[21088]: System set
| setsockopt, SO_RCVBUF to 262142 bytes
| May 12 08:35:33 localhost /usr/local/bin/nfcapd[21090]: Startup.
| May 12 08:35:33 localhost /usr/local/bin/nfcapd[21090]: Process_v9: New
| exporter domain 257
| May 12 08:35:33 localhost /usr/local/bin/nfcapd[21090]: Process_v9: [257]
| Add template 257
| May 12 08:35:33 localhost /usr/local/bin/nfcapd[21091]: Standard setsockopt,
| SO_RCVBUF is 110592 Requested length is 200000 bytes
| May 12 08:35:33 localhost /usr/local/bin/nfcapd[21091]: System set
| setsockopt, SO_RCVBUF to 262142 bytes
| May 12 08:35:33 localhost /usr/local/bin/nfcapd[21093]: Startup.
| May 12 08:35:33 localhost /usr/local/bin/nfcapd[21084]: Process_v9: New
| exporter domain 516
| May 12 08:35:33 localhost /usr/local/bin/nfcapd[21094]: Standard setsockopt,
| SO_RCVBUF is 110592 Requested length is 200000 bytes
| May 12 08:35:33 localhost /usr/local/bin/nfcapd[21094]: System set
| setsockopt, SO_RCVBUF to 262142 bytes
| May 12 08:35:33 localhost /usr/local/bin/nfcapd[21096]: Startup.
| May 12 08:35:33 localhost /usr/local/bin/nfcapd[21090]: Process_v9: New
| exporter domain 259
| May 12 08:35:33 localhost /usr/local/bin/nfcapd[21090]: Process_v9: [259]
| Add template 257
| May 12 08:35:33 localhost /usr/local/bin/nfcapd[21097]: Standard setsockopt,
| SO_RCVBUF is 110592 Requested length is 200000 bytes
| May 12 08:35:33 localhost /usr/local/bin/nfcapd[21097]: System set
| setsockopt, SO_RCVBUF to 262142 bytes
| May 12 08:35:34 localhost /usr/local/bin/nfcapd[21099]: Startup.
| May 12 08:35:34 localhost /usr/local/bin/nfcapd[21099]: Process_v9: New
| exporter domain 0
| May 12 08:35:34 localhost /usr/local/bin/nfcapd[21096]: Process_v9: New
| exporter domain 258
| May 12 08:35:34 localhost /usr/local/bin/nfcapd[21090]: Process_v9: New
| exporter domain 256
| May 12 08:35:34 localhost /usr/local/bin/nfcapd[21096]: Process_v9: [258]
| Add template 257
| May 12 08:35:34 localhost /usr/local/bin/nfcapd[21090]: Process_v9: [256]
| Add template 257
| May 12 08:35:34 localhost /usr/local/bin/nfcapd[21090]: Process_v9: New
| exporter domain 260
| May 12 08:35:34 localhost /usr/local/bin/nfcapd[21090]: Process_v9: [260]
| Add template 257
| May 12 08:35:34 localhost /usr/local/bin/nfcapd[21100]: Standard setsockopt,
| SO_RCVBUF is 110592 Requested length is 200000 bytes
| May 12 08:35:34 localhost /usr/local/bin/nfcapd[21100]: Standard setsockopt,
| SO_RCVBUF is 110592 Requested length is 200000 bytes
| May 12 08:35:34 localhost /usr/local/bin/nfcapd[21100]: System set
| setsockopt, SO_RCVBUF to 262142 bytes
| May 12 08:35:34 localhost /usr/local/bin/nfcapd[21102]: Startup.
| May 12 08:35:34 localhost /usr/local/bin/nfcapd[21102]: Process_v9: New
| exporter domain 258
| May 12 08:35:34 localhost /usr/local/bin/nfcapd[21096]: Process_v9: New
| exporter domain 257
| May 12 08:35:34 localhost /usr/local/bin/nfcapd[21096]: Process_v9: [257]
| Add template 257
| May 12 08:35:34 localhost /usr/local/bin/nfcapd[21102]: Process_v9: [258]
| Add template 256
| May 12 08:35:34 localhost /usr/local/bin/nfcapd[21103]: Standard setsockopt,
| SO_RCVBUF is 110592 Requested length is 200000 bytes
| May 12 08:35:34 localhost /usr/local/bin/nfcapd[21103]: System set
| setsockopt, SO_RCVBUF to 262142 bytes
| May 12 08:35:34 localhost /usr/local/bin/nfcapd[21105]: Startup.
| May 12 08:35:34 localhost /usr/local/bin/nfcapd[21096]: Process_v9: New
| exporter domain 259
| May 12 08:35:34 localhost /usr/local/bin/nfcapd[21106]: Standard setsockopt,
| SO_RCVBUF is 110592 Requested length is 200000 bytes
| May 12 08:35:34 localhost /usr/local/bin/nfcapd[21106]: System set
| setsockopt, SO_RCVBUF to 262142 bytes
| May 12 08:35:34 localhost /usr/local/bin/nfcapd[21108]: Startup.
| May 12 08:35:34 localhost /usr/local/bin/nfcapd[21096]: Process_v9: New
| exporter domain 256
| May 12 08:35:35 localhost nfsen[21109]: Startup. Version: 1.3 $Id: nfsend 22
| 2007-11-20 12:27:38Z phaag $
| May 12 08:35:35 localhost nfsen[21111]: Comm server started: [21111]
| May 12 08:35:35 localhost nfsen[21110]: nfsend: [21110]
| May 12 08:35:35 localhost nfsen[21111]: floodsearch BEGIN
| May 12 08:35:35 localhost floodsearch: Loading plugin 'floodsearch': Success
|
| May 12 08:35:35 localhost floodsearch: floodsearch: Init
| May 12 08:35:35 localhost floodsearch: Initializing plugin 'floodsearch':
| Success
| May 12 08:35:35 localhost floodsearch: plugin 'floodsearch': Profile plugin:
| 1, Alert condition plugin: 0, Alert action plugin: 0
| May 12 08:35:35 localhost floodsearch: prefixStats BEGIN
| May 12 08:35:35 localhost prefixStats: Loading plugin 'prefixStats': Success
|
| May 12 08:35:35 localhost prefixStats: prefixStats: Init
| May 12 08:35:35 localhost prefixStats: Initializing plugin 'prefixStats':
| Success
| May 12 08:35:35 localhost prefixStats: plugin 'prefixStats': Profile plugin:
| 1, Alert condition plugin: 0, Alert action plugin: 0
| May 12 08:35:35 localhost prefixStats: Frontend module 'PortTracker.php'
| found
| May 12 08:35:35 localhost prefixStats: Loading plugin 'PortTracker': Success
|
| May 12 08:35:35 localhost prefixStats: PortTracker: Init
| May 12 08:35:35 localhost prefixStats: Initializing plugin 'PortTracker':
| Success
| May 12 08:35:35 localhost prefixStats: plugin 'PortTracker': Profile plugin:
| 1, Alert condition plugin: 0, Alert action plugin: 0
| May 12 08:35:35 localhost prefixStats: Plugins for profile : ./live
| - floodsearch,prefixStats,PortTracker
| May 12 08:35:35 localhost prefixStats: No site specific lookup module found
| May 12 08:35:35 localhost prefixStats: Start to rebuild profile 'live'
| May 12 08:35:35 localhost /usr/local/bin/nfcapd[21096]: Process_v9: [259]
| Add template 257
| May 12 08:35:35 localhost /usr/local/bin/nfcapd[21096]: Process_v9: [256]
| Add template 256
| May 12 08:35:35 localhost /usr/local/bin/nfcapd[21084]: Process_v9: New
| exporter domain 515
| May 12 08:35:35 localhost /usr/local/bin/nfcapd[21084]: Process_v9: [515]
| Add template 256
| May 12 08:35:36 localhost nfsen[21110]: 0 channels/alerts to profile
| *May 12 08:35:36 localhost nfsen[21110]: Run expire at Mon May 12 08:35:00
| 2008
| May 12 08:35:36 localhost nfsen[21110]: End expire at Mon May 12 08:35:00
| 2008 *
|
| Logrotate contains the following:
|
| /var/log/floodsearch.log {
| # This file is a log generated by a nfsen plugin
| weekly
| create 0666 root root
| rotate 4
| postrotate
| /data/nfsen/bin/nfsen reload
| endscript
| }
|
| /var/log/prefixStats.log {
| # This file is a log generated by another nfsen plugin
| daily
| create 0666 root root
| rotate 2
| postrotate
| /data/nfsen/bin/nfsen reload
| endscript
| }
|
| Now, logrotate is run in cron.daily, which runs at 04:02. My expire problem
| appears only on weekends (on sunday morning), so I think there might be a
| problem when nfsen is reloaded twice in a short interval. The thing is that
| this doesn't happen every weekend!
|
| So, sorry for my long post, but my question is: how does the expire
| mechanism start? Is the expire run as a plugin? By the way, in the debug
| messages I don't get any more messages from my plugins (and I can confirm
| that my plugins don't run any longer after logwatch runs)!
|
| I guess a short-term solution would be to disable nfsen's restart after the
| weekly log is rotated, but this doesn't solve the core problem.
|
| If you need any more help in tracking down the problem, let me know...
|
| Thank you for your time and support,
| Adrian
|
|
|
|
| On Tue, Apr 22, 2008 at 1:48 PM, Adrian Popa <[EMAIL PROTECTED]>
| wrote:
|
| > Thank you for your reply.
| >
| > I searched the archive, and I found your answer:
| >
| >
| > *That's what you can define with the waterwark. Assume you have a size
| > limit
| > of 100MB and a watermark of 90%. Expiring data starts when the size is
| > over
| > the limit of 100MB. It will delete data down to 90% of 100MB = 90MB.
| > So the data size will oscillate between 90MB and 100MB. If you have
| > lifetime
| > limits the same applies here: 30days are 720h. 90% is 648h. So your data
| > oscillates between 648h and 720h.*
| > *The oscillation is mostly given by the level of the watermark you define
| > in
| > nfsen.conf.*
| >
| > So, from what you're saying, the used space would never go past the
| > profile size. Since I have only one profile (live), the only logical
| > explanation would be that I got ~20G of data for the whole profile in 5
| > minutes. This caused the disk to overflow and the cleaning mechanism didn't
| > manage to do its job.
| >
| > I find it very hard to believe that I get a spike in the number of flows
| > (of 20G) in just 5 minutes. I've looked over the traffic graphs for the
| > collector's interface and even if there is an increase in traffic compared
| > to week days, there is no such spike that would clarify the situation...
| >
| > I'll have to lower the maximum space for the profile, but this means it's
| > just unused space... :(
| >
| > Regards,
| > Adrian
| > On Tue, Apr 22, 2008 at 11:31 AM, Peter Haag <[EMAIL PROTECTED]> wrote:
| >
| > > -----BEGIN PGP SIGNED MESSAGE-----
| > > Hash: SHA1
| > >
| > > Adrian,
| > >
| > > - --On April 21, 2008 13:33:09 +0300 Adrian Popa <
| > > [EMAIL PROTECTED]> wrote:
| > >
| > > | Hello everybody,
| > > |
| > > | I ran into the same problem I had a while ago - nfsen sometimes
| > > (during the
| > > | weekend) runs out of disk space and is at a stand-still...
| > > |
| > > | Here's how my system looked this morning:
| > > |
| > > | [EMAIL PROTECTED] ~]# df -h
| > > | Filesystem Size Used Avail Use% Mounted on
| > > | /dev/sda1 15G 8.4G 5.4G 61% /
| > > | */dev/sda2 241G 229G 0 100% /data*
| > > | none 3.5G 0 3.5G 0% /dev/shm
| > > | */dev/sda5 13G 13G 0 100% /var*
| > > |
| > > | The /var partition was filled by the syslog error messages:
| > > | Apr 20 11:19:49 localhost /usr/local/bin/nfcapd[4490]: Process_v9:
| > > output
| > > | buffer size error. Abort v9 record processing
| > > | Apr 20 11:19:49 localhost /usr/local/bin/nfcapd[4487]: Failed to write
| > > | output buffer to disk: 'No space left on device'
| > > |
| > > | I have a script that checks for partitions overflowing, but for some
| > > reason
| > > | it didn't do the trick this time...
| > > |
| > > | After deleting some old flow data and clearing the /var partition, I
| > > rebuilt
| > > | the live profile and restarted nfsen...
| > > |
| > > | [EMAIL PROTECTED] clearOldRRDs]# ./emergencyCleanup.pl
| > > | /data 100
| > > | /var 100
| > > | Deleting files older than 2 days
| > > | /data 48
| > > | /var 85
| > > | [EMAIL PROTECTED] clearOldRRDs]# df -h
| > > | Filesystem Size Used Avail Use% Mounted on
| > > | /dev/sda1 15G 8.4G 5.4G 61% /
| > > | */dev/sda2 241G 110G 120G 48% /data*
| > > | none 3.5G 0 3.5G 0% /dev/shm
| > > | */dev/sda5 13G 11G 1.9G 85% /var*
| > > | [EMAIL PROTECTED] bin]# ./nfsen -r live
| > > | name live
| > > | group (nogroup)
| > > | tcreate Thu Apr 10 15:02:27 2008
| > > | tstart Thu Apr 17 23:40:00 2008
| > > | tend Mon Apr 21 09:25:00 2008
| > > | updated Mon Apr 21 09:25:00 2008
| > > | expire 3 days 0 hours
| > > | size 100.4 GB
| > > | *maxsize 200.0 GB*
| > > | type live
| > > | locked 0
| > > | status OK
| > > | version 130
| > > |
| > > |
| > > | My question is: if the maximum size of the profile is 200G, and the
| > > full
| > > | profile would occupy 210G, why did it fill my 230G partition?
| > >
| > > The channel size is updated from each collector as the files are stored
| > > on disk. The accumulated profile size is calculated by
| > > nfexpire at each 5min cycle run. If you manipulate anything by hand
| > > change/add/delete files, then you need to rebuild to total
| > > profile size, as NfSen has no ideas about any manual changes. Please
| > > also note, that NfSen handles profiles sizes on each
| > > individual bases and not according to the available size on a volume.
| > >
| > > |
| > > | Could you explain agian how the watermarking works? I would like to
| > > set it
| > > | to delete stuff as soon as possible and not be lazy...
| > >
| > > I posted an explanation a month ago to this list, don't know the exact
| > > date. It should be in the archive.
| > >
| > > - Peter
| > > |
| > > | Thank you,
| > > | Adrian
| > >
| > >
| > >
| > > - --
| > > _______ SWITCH - The Swiss Education and Research Network ______
| > > Peter Haag, Security Engineer, Member of SWITCH CERT
| > > PGP fingerprint: D9 31 D5 83 03 95 68 BA FB 84 CA 94 AB FC 5D D7
| > > SWITCH, Werdstrasse 2, P.O. Box, CH-8021 Zurich, Switzerland
| > > E-mail: [EMAIL PROTECTED] Web: http://www.switch.ch/
| > > -----BEGIN PGP SIGNATURE-----
| > > Version: GnuPG v1.4.3 (Darwin)
| > >
| > > iQCVAwUBSA2idv5AbZRALNr/AQKm7wP/cLP7M7cpeRCnRf3xIk8zZSRP6SBS40Wm
| > > aNala997gCka+XiuoUm7qD9asmE2vGqPiyXTJGxQSJn6Pzw5Qrts7IlmVXJ9UdVG
| > > 03/LXWdkH6xdBkDXoOXg1KsuBoSxZo4XbJ/eVEghOGhc9Qf8Qh9hYLVBEoMNxRhs
| > > SAZG5NLsV74=
| > > =ZGSr
| > > -----END PGP SIGNATURE-----
| > >
| > >
| >
- --
_______ SWITCH - The Swiss Education and Research Network ______
Peter Haag, Security Engineer, Member of SWITCH CERT
PGP fingerprint: D9 31 D5 83 03 95 68 BA FB 84 CA 94 AB FC 5D D7
SWITCH, Werdstrasse 2, P.O. Box, CH-8021 Zurich, Switzerland
E-mail: [EMAIL PROTECTED] Web: http://www.switch.ch/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (Darwin)
iQCVAwUBSCv7nv5AbZRALNr/AQJ0uQP+OSRbHqVh3Akjtbte4Yab6gi51q7ocLCC
gYhHIOh+oy2JSbuLysPpC3HWokQqRu/QqwjOECVfxxpyzSvfmg4fdtyxfZK1lTre
Pj2hsE0vXM4zTg9nSYvJ2L97+LGoSv0evxX3oIXTvlBFf4xgZJYZcvrMbbAM9k8Y
XmQDLCrIEto=
=5ji7
-----END PGP SIGNATURE-----
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
Nfsen-discuss mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
