Our new Graylog instance has been running for awhile without so much as a
hiccup. Recently, I added new log sources from a Security Onion sensor
containing BRO and Suricata logs. It doesn't appear these new inputs have
caused any noticeable load on the system, at least not until I run them
through a pipeline processor. I am using grok and standard regex functions
in the pipeline rules to parse out bro_conn, bro_dns, etc.
Today, I noticed the output stopped during what I would consider peak load
around 11:30a<ish> CST and to the best of my recollection no changes had
been made directly proceeding the stoppage.
As I have been adding pipeline rules to parse out messages, it seems
something happens where logs stop writing to the elasticsearch nodes. I
don't see anything in the server.log that looks like a smoking gun If I
restart the graylog-server.service the logs will not begin to clear from
the output buffer. However, If I stop the graylog-server.service and then
start it, logs begin to flow again. I do not have to restart any other
service after the manual stop/start of graylog.
The only log I see that seems like it would be related is below. However, I
am not sure if it is relevant.
2016-12-07T12:27:53.601-06:00 WARN [DeadEventLoggingListener] Received
unhandled event of type <org.graylog2.plugin.lifecycles.Lifecycle> from
event bus <AsyncEventBus{graylog-eventbus}>
top - 15:07:03 up 18 days, 22:46, 1 user, load average: 0.68, 1.06, 1.12
Tasks: 420 total, 1 running, 419 sleeping, 0 stopped, 0 zombie
%Cpu(s): 5.3 us, 1.6 sy, 0.0 ni, 93.0 id, 0.0 wa, 0.0 hi, 0.0 si,
0.0 st
KiB Mem : 49282804 total, 34329916 free, 11754760 used, 3198128 buff/cache
KiB Swap: 1048572 total, 1048572 free, 0 used. 37276428 avail Mem
Graylog 2.1.1+01d50e5 starting up
JRE: 1.8.0_111 on Linux 3.10.0-327.36.3.el7.x86_64
OS: CentOS Linux 7 (Core) amd64
JVM arguments: -Xms8g -Xmx8g -XX:NewRatio=1 -XX:+ResizeTLAB
-XX:+UseConcMarkSweepGC -XX:+CMSConc currentMTEnabled
-XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC
-XX:-OmitStackTraceInFastThrow
-Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml
-Djava.library.path=/opt/graylog-server/lib/sigar
-Dgraylog2.installation_source=unknown
transparent_hugepage=false
If someone could point me to a place where I can get better insight into
which pipeline rule(s) may be causing the problem, I would appreciate it. I
also am open to other suggestions too.
Regards,
Brandon
--
You received this message because you are subscribed to the Google Groups
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/graylog2/c48bebd2-adbb-4681-97c8-c70eb978af10%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
