Increasing the heap size on ES and Graylog respectively fixed the issue. On Friday, July 8, 2016 at 11:07:46 AM UTC-5, Ariel Godinez wrote: > > After further investigation I think this was due to elasticsearch and > graylog being overloaded. I have increased their heap sizes accordingly and > will see how the system performs. > > Ariel > > On Wednesday, July 6, 2016 at 12:21:11 PM UTC-5, Ariel Godinez wrote: >> >> Hello, >> >> I've been using graylog for a couple weeks now and started to notice some >> unusual behavior today. I am currently running a single node setup. >> >> The Issue: >> >> Every once in awhile I start to notice that that graylog is dragging >> quite a bit (the loading spinner is persisting much longer than usual) so I >> go check the logs and find the following error message. >> >> ERROR [ServerRuntime$Responder] An I/O error has occurred while writing a >> response message entity to the container output stream. >> org.glassfish.jersey.server.internal.process.MappableException: >> java.io.IOException: Connection closed >> at >> org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:92) >> >> ~[graylog.jar:?] >> at >> org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162) >> >> ~[graylog.jar:?] >> at >> org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1130) >> >> ~[graylog.jar:?] >> at >> org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:711) >> >> [graylog.jar:?] >> at >> org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:444) >> >> [graylog.jar:?] >> at >> org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:434) >> >> [graylog.jar:?] >> at >> org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:329) >> [graylog.jar:?] >> at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) >> [graylog.jar:?] >> at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) >> [graylog.jar:?] >> at org.glassfish.jersey.internal.Errors.process(Errors.java:315) >> [graylog.jar:?] >> at org.glassfish.jersey.internal.Errors.process(Errors.java:297) >> [graylog.jar:?] >> at org.glassfish.jersey.internal.Errors.process(Errors.java:267) >> [graylog.jar:?] >> at >> org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) >> >> [graylog.jar:?] >> at >> org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) >> [graylog.jar:?] >> at >> org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) >> >> [graylog.jar:?] >> at >> org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) >> >> [graylog.jar:?] >> at >> org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) >> [graylog.jar:?] >> at >> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) >> >> [graylog.jar:?] >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> >> [?:1.8.0_91] >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> >> [?:1.8.0_91] >> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91] >> Caused by: java.io.IOException: Connection closed >> at >> org.glassfish.grizzly.asyncqueue.TaskQueue.onClose(TaskQueue.java:317) >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter.onClose(AbstractNIOAsyncQueueWriter.java:501) >> >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.nio.transport.TCPNIOTransport.closeConnection(TCPNIOTransport.java:412) >> >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.nio.NIOConnection.doClose(NIOConnection.java:604) >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.nio.NIOConnection$5.run(NIOConnection.java:570) >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.nio.DefaultSelectorHandler.execute(DefaultSelectorHandler.java:235) >> >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.nio.NIOConnection.terminate0(NIOConnection.java:564) >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.nio.transport.TCPNIOConnection.terminate0(TCPNIOConnection.java:291) >> >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.nio.transport.TCPNIOAsyncQueueWriter.writeCompositeRecord(TCPNIOAsyncQueueWriter.java:197) >> >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.nio.transport.TCPNIOAsyncQueueWriter.write0(TCPNIOAsyncQueueWriter.java:92) >> >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter.processAsync(AbstractNIOAsyncQueueWriter.java:344) >> >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:107) >> >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77) >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:536) >> >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112) >> >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.strategies.SameThreadIOStrategy.executeIoEvent(SameThreadIOStrategy.java:103) >> >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.strategies.AbstractIOStrategy.executeIoEvent(AbstractIOStrategy.java:89) >> >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.nio.SelectorRunner.iterateKeyEvents(SelectorRunner.java:415) >> >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.nio.SelectorRunner.iterateKeys(SelectorRunner.java:384) >> >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:348) >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:279) >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:591) >> >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:571) >> >> ~[graylog.jar:?] >> ... 1 more >> Caused by: java.io.IOException: Broken pipe >> at sun.nio.ch.FileDispatcherImpl.write0(Native Method) >> ~[?:1.8.0_91] >> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) >> ~[?:1.8.0_91] >> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) >> ~[?:1.8.0_91] >> at sun.nio.ch.IOUtil.write(IOUtil.java:51) ~[?:1.8.0_91] >> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) >> ~[?:1.8.0_91] >> at >> org.glassfish.grizzly.nio.transport.TCPNIOUtils.flushByteBuffer(TCPNIOUtils.java:149) >> >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.nio.transport.TCPNIOAsyncQueueWriter.writeCompositeRecord(TCPNIOAsyncQueueWriter.java:187) >> >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.nio.transport.TCPNIOAsyncQueueWriter.write0(TCPNIOAsyncQueueWriter.java:92) >> >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.nio.AbstractNIOAsyncQueueWriter.processAsync(AbstractNIOAsyncQueueWriter.java:344) >> >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:107) >> >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:77) >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:536) >> >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:112) >> >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.strategies.SameThreadIOStrategy.executeIoEvent(SameThreadIOStrategy.java:103) >> >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.strategies.AbstractIOStrategy.executeIoEvent(AbstractIOStrategy.java:89) >> >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.nio.SelectorRunner.iterateKeyEvents(SelectorRunner.java:415) >> >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.nio.SelectorRunner.iterateKeys(SelectorRunner.java:384) >> >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.nio.SelectorRunner.doSelect(SelectorRunner.java:348) >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.nio.SelectorRunner.run(SelectorRunner.java:279) >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:591) >> >> ~[graylog.jar:?] >> at >> org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:571) >> >> ~[graylog.jar:?] >> ... 1 more >> >> At times I get a notification in the graylog web interface in conjunction >> with this error even though my *is_master = true *is indeed set : >> >> *There was no master Graylog server node detected in the >> cluster. (triggered a few seconds ago)* >> *Certain operations of Graylog server require the presence of a master >> node, but no such master was started. Please ensure that one of your >> Graylog server nodes contains the setting is_master = true in its >> configuration and that it is running. Until this is resolved index cycling >> will not be able to run, which means that the index retention mechanism is >> also not running, leading to increased index sizes. Certain maintenance >> functions as well as a variety of web interface pages (e.g. Dashboards) are >> unavailable.* >> >> After a couple minutes the notification just disappears and I continue to >> use graylog as normal. My thinking is that there is a communication issue >> somewhere along the tech stack. If anyone has any thoughts as to what might >> be causing the issue or where else I could look for error logs to further >> investigate please let me know. Thanks for your time. >> >> Regards, >> Ariel >> >>
-- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/f27e8c99-78d1-411c-9d81-939b9b54b7fe%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
