[graylog2] Re: Possible Cause Of Memory Leak Graylog 2(v2.1.2)

2016-12-08 Thread Yogesh Rao 
Hello Jochen!

Apologies for the delayed response...

This issue is being discussed in more detail 
@ https://github.com/Graylog2/graylog2-server/issues/3116

Please do let me know if you have any views on the same.

Thank you!

Regards,
-Yogesh

On Tuesday, December 6, 2016 at 10:05:35 PM UTC+5:30, Jochen Schalanda 
wrote:
>
> Hi Yogesh,
>
> please post your complete Graylog configuration file and describe your 
> setup in more detail.
>
> Cheers,
> Jochen
>
> On Friday, 25 November 2016 02:34:53 UTC+1, Yogesh Rao wrote:
>>
>> Hi,Github Issue # 3116 
>> 
>>
>> We have just started using graylog2 (2.1.2) on centos 7. I keep getting 
>> below exception after few hours
>>
>> 2016-11-24T17:25:13.737+05:30 WARN [ProxiedResource] Unable to call 
>> http://10.101.160.95:9000/api/system on node 
>>
>> java.net.SocketTimeoutException: timeout
>>
>> at okio.Okio$3.newTimeoutException(Okio.java:210) ~[graylog.jar:?]
>>
>> at okio.AsyncTimeout.exit(AsyncTimeout.java:288) ~[graylog.jar:?]
>>
>> at okio.AsyncTimeout$2.read(AsyncTimeout.java:242) ~[graylog.jar:?]
>>
>> at okio.RealBufferedSource.indexOf(RealBufferedSource.java:325) 
>> ~[graylog.jar:?]
>>
>> at okio.RealBufferedSource.indexOf(RealBufferedSource.java:314) 
>> ~[graylog.jar:?]
>>
>> at 
>> okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:210) 
>> ~[graylog.jar:?]
>>
>> at okhttp3.internal.http.Http1xStream.readResponse(Http1xStream.java:186) 
>> ~[graylog.jar:?]
>>
>> at 
>> okhttp3.internal.http.Http1xStream.readResponseHeaders(Http1xStream.java:127)
>>  
>> ~[graylog.jar:?]
>>
>> at 
>> okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:53)
>>  
>> ~[graylog.jar:?]
>>
>> at 
>> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
>>  
>> ~[graylog.jar:?]
>>
>> at 
>> okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45)
>>  
>> ~[graylog.jar:?]
>>
>> at 
>> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
>>  
>> ~[graylog.jar:?]
>>
>> at 
>> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
>>  
>> ~[graylog.jar:?]
>>
>> at 
>> okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:109) 
>> ~[graylog.jar:?]
>>
>> at 
>> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
>>  
>> ~[graylog.jar:?]
>>
>> at 
>> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
>>  
>> ~[graylog.jar:?]
>>
>> at 
>> okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) 
>> ~[graylog.jar:?]
>>
>> at 
>> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
>>  
>> ~[graylog.jar:?]
>>
>> at 
>> okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:124)
>>  
>> ~[graylog.jar:?]
>>
>> at 
>> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
>>  
>> ~[graylog.jar:?]
>>
>> at 
>> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
>>  
>> ~[graylog.jar:?]
>>
>> at 
>> org.graylog2.rest.RemoteInterfaceProvider.lambda$get$0(RemoteInterfaceProvider.java:59)
>>  
>> ~[graylog.jar:?]
>>
>> at 
>> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
>>  
>> ~[graylog.jar:?]
>>
>> at 
>> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
>>  
>> ~[graylog.jar:?]
>>
>> at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:170) 
>> ~[graylog.jar:?]
>>
>> at okhttp3.RealCall.execute(RealCall.java:60) ~[graylog.jar:?]
>>
>> at retrofit2.OkHttpCall.execute(OkHttpCall.java:174) ~[graylog.jar:?]
>>
>> at 
>> org.graylog2.shared.rest.resources.ProxiedResource.lambda$null$0(ProxiedResource.java:76)
>>  
>> ~[graylog.jar:?]
>>
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_102]
>>
>> at 
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>  
>> [?:1.8.0_102]
>>
>> at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>  
>> [?:1.8.0_102]
>>
>> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_102]
>>
>> Caused by: java.net.SocketTimeoutException: Read timed out
>>
>> at java.net.SocketInputStream.socketRead0(Native Method) ~[?:1.8.0_102]
>>
>> at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) 
>> ~[?:1.8.0_102]
>>
>> at java.net.SocketInputStream.read(SocketInputStream.java:170) 
>> ~[?:1.8.0_102]
>>
>> at java.net.SocketInputStream.read(SocketInputStream.java:141) 
>> ~[?:1.8.0_102]
>>
>> at okio.Okio$2.read(Okio.java:138) ~[graylog.jar:?]
>>
>> at okio.AsyncTimeout$2.read(AsyncTimeout.java:238) ~[graylog.jar:?]
>>
>> ... 29 more
>>
>> 2016-11-24T17:25:13.751+05:30 WARN [jvm] 
>> [graylog-b8d88e71-dbbb-44ee-9951-2bcdb45b047a] [gc][old][19743][529] 
>> duration [12.4s],

[graylog2] Re: Possible Cause Of Memory Leak Graylog 2(v2.1.2)

2016-12-06 Thread Jochen Schalanda 
Hi Yogesh,

please post your complete Graylog configuration file and describe your 
setup in more detail.

Cheers,
Jochen

On Friday, 25 November 2016 02:34:53 UTC+1, Yogesh Rao wrote:
>
> Hi,
>
> We have just started using graylog2 (2.1.2) on centos 7. I keep getting 
> below exception after few hours
>
> 2016-11-24T17:25:13.737+05:30 WARN [ProxiedResource] Unable to call 
> http://10.101.160.95:9000/api/system on node 
>
> java.net.SocketTimeoutException: timeout
>
> at okio.Okio$3.newTimeoutException(Okio.java:210) ~[graylog.jar:?]
>
> at okio.AsyncTimeout.exit(AsyncTimeout.java:288) ~[graylog.jar:?]
>
> at okio.AsyncTimeout$2.read(AsyncTimeout.java:242) ~[graylog.jar:?]
>
> at okio.RealBufferedSource.indexOf(RealBufferedSource.java:325) 
> ~[graylog.jar:?]
>
> at okio.RealBufferedSource.indexOf(RealBufferedSource.java:314) 
> ~[graylog.jar:?]
>
> at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:210) 
> ~[graylog.jar:?]
>
> at okhttp3.internal.http.Http1xStream.readResponse(Http1xStream.java:186) 
> ~[graylog.jar:?]
>
> at 
> okhttp3.internal.http.Http1xStream.readResponseHeaders(Http1xStream.java:127) 
> ~[graylog.jar:?]
>
> at 
> okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:53)
>  
> ~[graylog.jar:?]
>
> at 
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
>  
> ~[graylog.jar:?]
>
> at 
> okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45)
>  
> ~[graylog.jar:?]
>
> at 
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
>  
> ~[graylog.jar:?]
>
> at 
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
>  
> ~[graylog.jar:?]
>
> at 
> okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:109) 
> ~[graylog.jar:?]
>
> at 
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
>  
> ~[graylog.jar:?]
>
> at 
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
>  
> ~[graylog.jar:?]
>
> at 
> okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) 
> ~[graylog.jar:?]
>
> at 
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
>  
> ~[graylog.jar:?]
>
> at 
> okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:124)
>  
> ~[graylog.jar:?]
>
> at 
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
>  
> ~[graylog.jar:?]
>
> at 
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
>  
> ~[graylog.jar:?]
>
> at 
> org.graylog2.rest.RemoteInterfaceProvider.lambda$get$0(RemoteInterfaceProvider.java:59)
>  
> ~[graylog.jar:?]
>
> at 
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
>  
> ~[graylog.jar:?]
>
> at 
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
>  
> ~[graylog.jar:?]
>
> at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:170) 
> ~[graylog.jar:?]
>
> at okhttp3.RealCall.execute(RealCall.java:60) ~[graylog.jar:?]
>
> at retrofit2.OkHttpCall.execute(OkHttpCall.java:174) ~[graylog.jar:?]
>
> at 
> org.graylog2.shared.rest.resources.ProxiedResource.lambda$null$0(ProxiedResource.java:76)
>  
> ~[graylog.jar:?]
>
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_102]
>
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  
> [?:1.8.0_102]
>
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  
> [?:1.8.0_102]
>
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_102]
>
> Caused by: java.net.SocketTimeoutException: Read timed out
>
> at java.net.SocketInputStream.socketRead0(Native Method) ~[?:1.8.0_102]
>
> at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) 
> ~[?:1.8.0_102]
>
> at java.net.SocketInputStream.read(SocketInputStream.java:170) 
> ~[?:1.8.0_102]
>
> at java.net.SocketInputStream.read(SocketInputStream.java:141) 
> ~[?:1.8.0_102]
>
> at okio.Okio$2.read(Okio.java:138) ~[graylog.jar:?]
>
> at okio.AsyncTimeout$2.read(AsyncTimeout.java:238) ~[graylog.jar:?]
>
> ... 29 more
>
> 2016-11-24T17:25:13.751+05:30 WARN [jvm] 
> [graylog-b8d88e71-dbbb-44ee-9951-2bcdb45b047a] [gc][old][19743][529] 
> duration [12.4s], collections [1]/[12.6s], total [12.4s]/[10.4m], memory 
> [3.7gb]->[3.7gb]/[3.8gb], all_pools {[young] 
> [1.6gb]->[1.6gb]/[1.6gb]}{[survivor] [188.7mb]->[191.2mb]/[204.7mb]}{[old] 
> [1.9gb]->[1.9gb]/[2gb]}
> 2016-11-24T17:25:13.821+05:30 INFO [BeatsCodec] inside 
> decodeMessagesRawMessage{id=52182873-b20e-11e6-9b98-14feb5ea6076, 
> journalOffset=960249, codec=beats, payloadSize=4169, 
> timestamp=2016-11-24T06:22:05.798Z, remoteAddress=/10.101.160.6:49324}
>
>
> Heap Size set for Graylog2 is 4G, I noticed the heap size jumps up from 
> 1.5G to 4G as soon as the exceptions are