Hi,
We have just started using graylog2 (2.1.2) on centos 7. I keep getting below exception after few hours 2016-11-24T17:25:13.737+05:30 WARN [ProxiedResource] Unable to call http://10.101.160.95:9000/api/system on node java.net.SocketTimeoutException: timeout at okio.Okio$3.newTimeoutException(Okio.java:210) ~[graylog.jar:?] at okio.AsyncTimeout.exit(AsyncTimeout.java:288) ~[graylog.jar:?] at okio.AsyncTimeout$2.read(AsyncTimeout.java:242) ~[graylog.jar:?] at okio.RealBufferedSource.indexOf(RealBufferedSource.java:325) ~[graylog.jar:?] at okio.RealBufferedSource.indexOf(RealBufferedSource.java:314) ~[graylog.jar:?] at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:210) ~[graylog.jar:?] at okhttp3.internal.http.Http1xStream.readResponse(Http1xStream.java:186) ~[graylog.jar:?] at okhttp3.internal.http.Http1xStream.readResponseHeaders(Http1xStream.java:127) ~[graylog.jar:?] at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:53) ~[graylog.jar:?] at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?] at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45) ~[graylog.jar:?] at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?] at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?] at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:109) ~[graylog.jar:?] at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?] at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?] at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[graylog.jar:?] at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?] at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:124) ~[graylog.jar:?] at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?] at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?] at org.graylog2.rest.RemoteInterfaceProvider.lambda$get$0(RemoteInterfaceProvider.java:59) ~[graylog.jar:?] at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?] at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?] at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:170) ~[graylog.jar:?] at okhttp3.RealCall.execute(RealCall.java:60) ~[graylog.jar:?] at retrofit2.OkHttpCall.execute(OkHttpCall.java:174) ~[graylog.jar:?] at org.graylog2.shared.rest.resources.ProxiedResource.lambda$null$0(ProxiedResource.java:76) ~[graylog.jar:?] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_102] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_102] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_102] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_102] Caused by: java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) ~[?:1.8.0_102] at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) ~[?:1.8.0_102] at java.net.SocketInputStream.read(SocketInputStream.java:170) ~[?:1.8.0_102] at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_102] at okio.Okio$2.read(Okio.java:138) ~[graylog.jar:?] at okio.AsyncTimeout$2.read(AsyncTimeout.java:238) ~[graylog.jar:?] ... 29 more 2016-11-24T17:25:13.751+05:30 WARN [jvm] [graylog-b8d88e71-dbbb-44ee-9951-2bcdb45b047a] [gc][old][19743][529] duration [12.4s], collections [1]/[12.6s], total [12.4s]/[10.4m], memory [3.7gb]->[3.7gb]/[3.8gb], all_pools {[young] [1.6gb]->[1.6gb]/[1.6gb]}{[survivor] [188.7mb]->[191.2mb]/[204.7mb]}{[old] [1.9gb]->[1.9gb]/[2gb]} 2016-11-24T17:25:13.821+05:30 INFO [BeatsCodec] inside decodeMessagesRawMessage{id=52182873-b20e-11e6-9b98-14feb5ea6076, journalOffset=960249, codec=beats, payloadSize=4169, timestamp=2016-11-24T06:22:05.798Z, remoteAddress=/10.101.160.6:49324} Heap Size set for Graylog2 is 4G, I noticed the heap size jumps up from 1.5G to 4G as soon as the exceptions are encountered, eventually leading to slower GC's and finally ends up unresponsive. All the buffer processors are set to 1 process_buffer and output buffer are set to 32768 Data pumped into graylog is around 4 GB Graylog is configured on a Core 2 Duo machine with 8GB Ram and is pumping data into 3 elasticsearch (clustered) instances. How can i troubleshoot this issue to figure out what the actual problem is ? -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/8e7ef2d9-cbf0-4259-b8a4-e885ab9c9cba%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
