I'm very new to all of this and have been running graylog for only the past week - all three components on one box. It's been working well until last night when elasticsearch crashed/exited. The /var/log/elasticsearch/xx.log file contains no error, but syslog said
Jul 30 15:35:06 gl2.local graylog-server: Exception in thread "elasticsearch[graylog2-server][[http_server_worker.default]][T#5]" Exception in thread "elasticsearch[graylog2-server][generic][T#252]" Jul 30 15:35:06 gl2.local graylog-server: Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "elasticsearch[graylog2-server][generic][T#251]" Jul 30 15:35:40 gl2.local graylog-server: java.lang.OutOfMemoryError: Java heap space Jul 30 15:35:40 gl2.local graylog-server: java.lang.OutOfMemoryError: Java heap space Jul 30 15:36:42 gl2.local graylog-server: Exception in thread "metrics-meter-tick-thread-1" java.lang.OutOfMemoryError: Java heap space Jul 30 15:56:04 gl2.local graylog-server: Exception in thread "restapi-worker-11" java.lang.OutOfMemoryError: Java heap space Jul 30 15:56:27 gl2.local graylog-server: Exception in thread "metrics-meter-tick-thread-2" java.lang.OutOfMemoryError: Java heap space Jul 30 15:58:26 gl2.local graylog-server: Exception in thread "restapi-boss-0" java.lang.OutOfMemoryError: Java heap space Jul 30 16:00:54 gl2.local graylog-server: Exception in thread "processbufferprocessor-4" java.lang.OutOfMemoryError: Java heap space Jul 30 16:00:55 gl2.local graylog-server: Exception in thread "processbufferprocessor-0" java.lang.OutOfMemoryError: Java heap space The odd thing was that top didn't show the system was out of memory - so I guess that hit one of the memory limits java was configured for - but that is meant to limit how much RAM it uses - not make it crash right? Anyway, when I noticed that, I rebooted - to ensure everything was happy - but no new data is flowing in. Elasticsearch looks happy, graylog-web looks happy - but the problem must be with graylog-server? It's logfile contains the following non-INFO lines 2015-07-30T16:23:44.562-04:00 WARN [discovery] [graylog2-server] waited for 3s and no initial state was set by the discovery 2015-07-30T16:23:47.575-04:00 ERROR [KafkaJournal] Read offset 0 before start of log at 188654530, starting to read from the beginning of the journal. 2015-07-30T16:23:47.625-04:00 WARN [NettyTransport] receiveBufferSize (SO_RCVBUF) for input org.graylog2.inputs.gelf.tcp.GELFTCPInput@f070f8e7 should be 1048576 but is 212992. 2015-07-30T16:23:47.657-04:00 WARN [NettyTransport] receiveBufferSize (SO_RCVBUF) for input org.graylog2.inputs.syslog.tcp.SyslogTCPInput@5b9d6e55 should be 1048576 but is 212992. 2015-07-30T16:23:47.671-04:00 WARN [NettyTransport] receiveBufferSize (SO_RCVBUF) for input org.graylog2.inputs.syslog.udp.SyslogUDPInput@30ac7673 should be 262144 but is 212992. 2015-07-30T16:23:47.674-04:00 WARN [NettyTransport] receiveBufferSize (SO_RCVBUF) for input org.graylog2.inputs.syslog.tcp.SyslogTCPInput@ffadcd14 should be 1048576 but is 212992. 2015-07-30T16:38:22.630-04:00 WARN [KafkaJournal] Committing journal offset spins 10 times now, this might be a bug. Continuing to try update. 2015-07-30T16:39:01.910-04:00 WARN [KafkaJournal] Committing journal offset spins 10 times now, this might be a bug. Continuing to try update. 2015-07-30T16:39:33.138-04:00 WARN [KafkaJournal] Committing journal offset spins 10 times now, this might be a bug. Continuing to try update. I was originally running 1.14, but two days ago upgraded to 1.15. That seemed to work fine (rebooted it after the upgrade too), but that "try upgrade" comment there makes me wonder if something went wrong? Through graylog-web I can still run searches - but not on new data. That implies to me elasticpath is now happy Help? Thanks Jason -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/ee3c2460-ef20-4fa5-b5e3-b5408bd44a77%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
