[graylog2] Re: Output rate very low with graylog 2.0rc1

[oliv . teltel]

And I also upgraded to graylog 2.0GA

Le lundi 2 mai 2016 11:01:10 UTC+2, oliv@gmail.com a écrit :
>
> I've just upgraded the ram and cpu of all the nodes but I still got the 
> same issue. Process and output buffers are at 100% and the journal keeps on 
> filling.
> Here is the new configuration:
> graylog server with 9 vcpu and 8 gig of ram (4gig allocated to graylog)
> elasticsearch nodes with 4 vcpu and 8 gig of ram (4 gig allocated to 
> elasticsearch)
>
> Is there any metrics that could help to find where the bottleneck is ?
>
> Thanks !
>
> Le mardi 26 avril 2016 16:22:12 UTC+2, Peter Krammer a écrit :
>>
>> Based on that both the Process and the Output buffers are full, I would 
>> say your elasticsearch nodes are not fast enough. 
>> They seem undersized in my eyes and should have more CPU and RAM. 
>> Also check if you set your -Xms and -Xmx settings (for both graylog and 
>> elasticsearch) to reasonable values, it helped me alot to increase them to 
>> half of my RAM size. (eg. -Xms8g -Xmx8g )
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/eef2ec91-24c7-4b18-90b3-e7dc54b7c662%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Output rate very low with graylog 2.0rc1

[oliv . teltel]

I've just upgraded the ram and cpu of all the nodes but I still got the 
same issue. Process and output buffers are at 100% and the journal keeps on 
filling.
Here is the new configuration:
graylog server with 9 vcpu and 8 gig of ram (4gig allocated to graylog)
elasticsearch nodes with 4 vcpu and 8 gig of ram (4 gig allocated to 
elasticsearch)

Is there any metrics that could help to find where the bottleneck is ?

Thanks !

Le mardi 26 avril 2016 16:22:12 UTC+2, Peter Krammer a écrit :
>
> Based on that both the Process and the Output buffers are full, I would 
> say your elasticsearch nodes are not fast enough. 
> They seem undersized in my eyes and should have more CPU and RAM. 
> Also check if you set your -Xms and -Xmx settings (for both graylog and 
> elasticsearch) to reasonable values, it helped me alot to increase them to 
> half of my RAM size. (eg. -Xms8g -Xmx8g )
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/926e74ae-4a55-4786-807c-da64b0396149%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Output rate very low with graylog 2.0rc1

[oliv . teltel]

I have only one GELF input on each node and no extractors.
I forgot to mention that logstash is also running on each node, it receives 
all the message, does some filtering then sends the messages to graylog via 
gelf. I've just disabled all the logstash filters but still have the 
performance issue 
The thing is that everything was working fine with graylog 1.3 and the 
exact same setup, the performance problems started just after upgrading to 
2.0

Thanks

Le vendredi 22 avril 2016 16:47:22 UTC+2, Jochen Schalanda a écrit :
>
> Hi,
>
> what kind of inputs are you running and are you using any extractors or 
> the message processing pipeline?
>
> A full process buffer usually hints to slow/complex message filtering or 
> extractors.
>
> Cheers,
> Jochen
>
> On Friday, 22 April 2016 16:29:41 UTC+2, oliv@gmail.com wrote:
>>
>> Hi,
>> I have a problem with my output rate which is very low. My input rate is 
>> not very high (200-400 msg/sec) but the output gets stuck at 100msg/sec 
>> max. As you can see in the following screenshot, the process and output 
>> buffers are at 100% and the journal is pretty full.
>>
>>
>> Here is my hardware setup :
>>
>> - 2 graylog servers. Both nodes have 4g of ram and 2vcpu, I've tried 
>> increasing it to 4vcpu and 8g of ram but ended up with the same problem
>>
>> - 3 elasticsearch nodes with each 1vcpu and 2 g of ram, 
>>
>> I have never experienced performance issues with graylog 1.x and the same 
>> input rate
>>
>>
>> In the logs I can see errors like the following
>> ERROR [KafkaJournal] Read offset 91635221 before start of log at 
>> 91753414, starting to read from the beginning of the journal.
>>
>>
>> Thanks !
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/a1a04789-b326-4a55-9e89-3e8aaf350952%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Output rate very low with graylog 2.0rc1

[oliv . teltel]

Hi,
I have a problem with my output rate which is very low. My input rate is 
not very high (200-400 msg/sec) but the output gets stuck at 100msg/sec 
max. As you can see in the following screenshot, the process and output 
buffers are at 100% and the journal is pretty full.

[image: graylog] 


Here is my hardware setup :

- 2 graylog servers. Both nodes have 4g of ram and 2vcpu, I've tried 
increasing it to 4vcpu and 8g of ram but ended up with the same problem

- 3 elasticsearch nodes with each 1vcpu and 2 g of ram, 

I have never experienced performance issues with graylog 1.x and the same 
input rate


In the logs I can see errors like the following
ERROR [KafkaJournal] Read offset 91635221 before start of log at 91753414, 
starting to read from the beginning of the journal.


Thanks !

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/141f8f62-6a27-469f-aa76-6ff95a388d41%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.