Luca,
I filed the bug on bugzilla.ntop.org.
Here is how I start snort 2.8.6.1 and snort 2.9.0.5. Both versions
frequently lock up the same way when the group of snort daemons on a
given box are restarted.
taskset $CPUMASK snort -i $IFACE -l
/opt/nids/sensor/var/log/snort/$IFACE -u snort -c
/opt/nids/sensor/etc/snort/rules/$IFACE/snort.conf --create-pidfile
--pid-path /var/run/snort -D -F /opt/nids/sensor/etc/snort/$IFACE.bpf
I have not yet tried snort clustering yet.
My modprobe config line for pf_ring looks like this:
options pf_ring num_slots=30000 transparent_mode=0 enable_tx_capture=0
Kevin Branch
On 6/16/2011 1:08 PM, Luca Deri wrote:
Kevin
it looks to me that there's a lock that is not released and that causes this
problem. it might be that is the clustering you use.
Can you please explain how you started snort?
Please file a bug on bugzilla.ntop.org for tracking this issue
Regards Luca
On Jun 16, 2011, at 6:53 PM, Kevin Branch wrote:
Ever since upgrading a couple of my CentOS 4.6 NIDS sensor hosts to use the
latest PF_RING from subversion, they have been recurrently locking up with CPU
soft lockup errors like this, forcing a manual system reboot:
BUG: soft lockup - CPU#1 stuck for 61s! [snort:5049]
On one sensor host, I downgraded to PF_RING 4195 and the problem completely
went away. However on the other host I am needing to take advantage of the
snort 2.9 daq and pf_ring clustering, so in that case I replaced the hardware
(now a dual-core hyperthreaded Xeon system) and switched to Ubuntu 10.04 server
in hopes of getting better alignment with current PF_RING development and
eliminating this problem -- but it still comes up frequently. I've also
diagnostically downgraded from snort-2.9.0.5 to snort 2.8.6.1 and shut down all
other NIDS processes besides snort.
Currently on my Ubuntu sensor host, I am running 4 instances of snort 2.8.6.1
linked against the latest PF_RING. Each snort instance is sniffing a unique
physical network interface (e1000). I am not using transparent mode or the
PF_RING version of the e1000 driver at this point. When I shut down these
snort instances and then attempt to start them up again, at least half the time
the NIDS host seizes up with CPU soft lockup errors and has to be rebooted. I
have tried disabling hyperthreading to no avail. I also started using
taskset to set cpu affinity for each snort process such that each instance runs
on a unique logical cpu. I have been able to recreate this problem even when
limiting myself to 2 snort processes, as long as they are running on the same
physical cpu.
When I rebuild snort 2.8.6.1 to not use PF_RING, I can restart all 4 snort
instances endlessly without CPU soft lockups.
It seems that I can't reliably restart sets of snort daemons on the same
physical CPU if snort is linked against current PF_RING.
I really don't know where to go from here, so I'm hoping someone else has
encountered something like this or can suggest where I should go next in the
process of diagnosing the issue.
Kevin Branch
_______________________________________________
Ntop-misc mailing list
[email protected]
http://listgateway.unipi.it/mailman/listinfo/ntop-misc
---
If you can not measure it, you can not improve it - Lord Kelvin
_______________________________________________
Ntop-misc mailing list
[email protected]
http://listgateway.unipi.it/mailman/listinfo/ntop-misc
_______________________________________________
Ntop-misc mailing list
[email protected]
http://listgateway.unipi.it/mailman/listinfo/ntop-misc
