[graylog2] Re: Graylog cant handle large amounts of incoming logs

[Arie]

Like the config file tell's, you could increase the 
processbuffer_processors and / or outputbuffer_processors 
if your buffers are filling up. Keep an eye on processor resources.

# The number of parallel running processors.
# Raise this number if your buffers are filling up.
processbuffer_processors = 5
outputbuffer_processors = 3

On Monday, November 23, 2015 at 10:55:48 AM UTC+1, Matthew Simon wrote:
>
> Hi Guys 
>
>
> I have a problem!
>
>
> I receive large amounts of logs to my Graylog2 server and i feel that the 
> server cant keep up with the incoming logs, Is there a way that I can 
> optimize my configuration to handle large amounts of LOGS. 
>
>
> Please see the image bellow.
>
>
> Thanks in advance.
>
>
>
>
> 
>
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/fec911dc-910c-4e16-8caa-ab1a09849385%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Graylog cant handle large amounts of incoming logs

[Emerson Coimbra]

Hi Matthew,

I have used Graylog in my job about a month and everything was going fine 
with the following setup:

Graylog-server and Graylog-web Graylog in a single VM (ESXi 5.5, Dell R910) 
with
2 cores
4GB RAM
20GB HDD
Graylog-server JVM 2GB
Graylog-webJVM default
Elasticsearch  JVM 256MB


Elasticsearch cluster with 3 nodes on 3 Dell T320
4 cores
8GB RAM
HDD 1TB SATA
Elasticsearch   JVM 6GB

**All servers are running CentOS 7.1 and OpenJDK 8. All packages were 
installed with repositories.

I have a lot of inputs, extractors and dashboards.

I'm collecting logs of some servers and some network assets. The device who 
generates more log is our Cisco ASA 5520. I configured the ASA to send logs 
from level 5 (notification) because level 6 (informational) generates a lot 
of unnecessary information. Yesterday my boss asked to measure ASA traffic 
in Graylog and I started to collect four new event level 6 (built and 
teardown, tcp / udp).
We jump from 2.5k messages per minute to about 20k. The load average of VM 
jumped above 9.0, hours later. At a given time, I had 32k unprocessed 
messages in journal queue, 

At night, I turned off the VM, increase its setting to 4 core and start it 
again. Now, everything is run like a charm with a load average between 3 
and 4.

Cheers,
Emerson

On Monday, November 23, 2015 at 7:55:48 AM UTC-2, Matthew Simon wrote:
>
> Hi Guys 
>
>
> I have a problem!
>
>
> I receive large amounts of logs to my Graylog2 server and i feel that the 
> server cant keep up with the incoming logs, Is there a way that I can 
> optimize my configuration to handle large amounts of LOGS. 
>
>
> Please see the image bellow.
>
>
> Thanks in advance.
>
>
>
>
> 
>
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/7e3954fd-2eec-463d-b0a4-df5378ab4e86%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Graylog cant handle large amounts of incoming logs

[HockeyFan0000]

 Initially, I used the OVA as well and I got horrendous performance.  I 
pointed only two file servers and two domain controllers to that 
installation and the VM was so busy the web interface would hang 
constantly.  I created another VM and installed Graylog using these 
instructions 

 
and got great performance, even with the same memory and CPU core count.  
Once I got Graylog working on the VM, I used the same instructions to 
install it on a physical server.  So far, I've collected almost 400 million 
log events in 11 days and the highest I've seen the CPU usage has been 14% 
with only 16GB of RAM.  My suggestion is to ditch the OVA altogether.

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/aab67b34-d5a6-4b4c-9957-24d7ad1aa566%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Graylog cant handle large amounts of incoming logs

[HockeyFan0000]

Initially, I used the OVA as well and I got horrendous performance.  I 
pointed only two file servers and two domain controllers to that 
installation and the VM was so busy the web interface would hang 
constantly.  I created another VM and installed Graylog using these 
instructions 

 
and got great performance, even with the same memory and CPU core count.  
Once I got Graylog working on the VM, I used the same instructions to 
install it on a physical server.  So far, I've collected almost 400 million 
log events in 11 days and the highest I've seen the CPU usage has been 14% 
with only 16GB of RAM.  My suggestion is to ditch the OVA altogether.





On Monday, November 23, 2015 at 6:38:50 AM UTC-5, Matthew Simon wrote:
>
> Hi Jochen 
>
> I installed Graylog2 via the OVA I downloaded off the Site.
>
> It can fluctuate you see there are specials that run on the website which 
> can cause massive amounts of traffic so it can be fine one week then within 
> the span of 2 days I can get anything from 1million messages to 4 million 
> messages. Common story is that it cant keep up with that influx of messages.
>
> Grok patterns that I use. Please see attachment Grokpatternsfile.txt
>
> Extractors please see attachment Extractors.txt
>
> I have not setup dashboards or streams as of yet.
>
> Purpose of Graylog is im trying to extract WAF logs (so if there is a 
> better way of doing this please help.)
>
> If there is more info you want please let me know and I do appreciate your 
> help in advance.
>
> Thanks 
>
>
> On Monday, November 23, 2015 at 12:58:45 PM UTC+2, Jochen Schalanda wrote:
>>
>> Hi Matthew,
>>
>> to not duplicate the information you already gave on Twitter (
>> https://twitter.com/Malfufi/status/668724729629556736):
>>
>>> heap 1.4G (changeability?) no local configuration has been made to 
>>> optimize,85mill and CPU 100% RAM 100%
>>
>>
>> The heap size of Graylog can be changed quite easily. How did you install 
>> Graylog in the first place (OVA, DEB packages, RPM packages, tarball…)?
>>
>> "85 mill" what exactly? Messages per second/minute/hour/day/week/month? 
>> Are you running any extractors (e. g. RegEx or Grok extractors) or defined 
>> streams with relatively complicated regular expression rules?
>>
>>
>> Cheers,
>> Jochen
>>
>> On Monday, 23 November 2015 10:55:48 UTC+1, Matthew Simon wrote:
>>>
>>> Hi Guys 
>>>
>>>
>>> I have a problem!
>>>
>>>
>>> I receive large amounts of logs to my Graylog2 server and i feel that 
>>> the server cant keep up with the incoming logs, Is there a way that I can 
>>> optimize my configuration to handle large amounts of LOGS. 
>>>
>>>
>>> Please see the image bellow.
>>>
>>>
>>> Thanks in advance.
>>>
>>>
>>>
>>>
>>> 
>>>
>>>
>>>
>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/a2179e59-3d64-48f3-9a62-a395f7d14751%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Graylog cant handle large amounts of incoming logs

[Jochen Schalanda]

Hi Matthew,

to not duplicate the information you already gave on Twitter (
https://twitter.com/Malfufi/status/668724729629556736):

> heap 1.4G (changeability?) no local configuration has been made to 
> optimize,85mill and CPU 100% RAM 100%


The heap size of Graylog can be changed quite easily. How did you install 
Graylog in the first place (OVA, DEB packages, RPM packages, tarball…)?

"85 mill" what exactly? Messages per second/minute/hour/day/week/month? Are 
you running any extractors (e. g. RegEx or Grok extractors) or defined 
streams with relatively complicated regular expression rules?


Cheers,
Jochen

On Monday, 23 November 2015 10:55:48 UTC+1, Matthew Simon wrote:
>
> Hi Guys 
>
>
> I have a problem!
>
>
> I receive large amounts of logs to my Graylog2 server and i feel that the 
> server cant keep up with the incoming logs, Is there a way that I can 
> optimize my configuration to handle large amounts of LOGS. 
>
>
> Please see the image bellow.
>
>
> Thanks in advance.
>
>
>
>
> 
>
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/9521f8e0-cde7-4d9a-9d37-12c57506490d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Graylog cant handle large amounts of incoming logs

[Matthew Simon]

Hi Jochen 

I installed Graylog2 via the OVA I downloaded off the Site.

It can fluctuate you see there are specials that run on the website which 
can cause massive amounts of traffic so it can be fine one week then within 
the span of 2 days I can get anything from 1million messages to 4 million 
messages. Common story is that it cant keep up with that influx of messages.

Grok patterns that I use. Please see attachment Grokpatternsfile.txt

Extractors please see attachment Extractors.txt

I have not setup dashboards or streams as of yet.

Purpose of Graylog is im trying to extract WAF logs (so if there is a 
better way of doing this please help.)

If there is more info you want please let me know and I do appreciate your 
help in advance.

Thanks 


On Monday, November 23, 2015 at 12:58:45 PM UTC+2, Jochen Schalanda wrote:
>
> Hi Matthew,
>
> to not duplicate the information you already gave on Twitter (
> https://twitter.com/Malfufi/status/668724729629556736):
>
>> heap 1.4G (changeability?) no local configuration has been made to 
>> optimize,85mill and CPU 100% RAM 100%
>
>
> The heap size of Graylog can be changed quite easily. How did you install 
> Graylog in the first place (OVA, DEB packages, RPM packages, tarball…)?
>
> "85 mill" what exactly? Messages per second/minute/hour/day/week/month? 
> Are you running any extractors (e. g. RegEx or Grok extractors) or defined 
> streams with relatively complicated regular expression rules?
>
>
> Cheers,
> Jochen
>
> On Monday, 23 November 2015 10:55:48 UTC+1, Matthew Simon wrote:
>>
>> Hi Guys 
>>
>>
>> I have a problem!
>>
>>
>> I receive large amounts of logs to my Graylog2 server and i feel that the 
>> server cant keep up with the incoming logs, Is there a way that I can 
>> optimize my configuration to handle large amounts of LOGS. 
>>
>>
>> Please see the image bellow.
>>
>>
>> Thanks in advance.
>>
>>
>>
>>
>> <https://lh3.googleusercontent.com/-n91LHSZ_HtQ/VlLhwvs-sKI/AqU/dXnmzSAQFCg/s1600/image1.png>
>>
>>
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/5137a75c-31d7-4040-b0b1-18e836d191cd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
USERNAME [a-zA-Z0-9._-]+
USER %{USERNAME}
INT (?:[+-]?(?:[0-9]+))
BASE10NUM (?[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+)))
NUMBER (?:%{BASE10NUM})
BASE16NUM (?(?"(?>\\.|[^\\"]+)+"|""|(?>'(?>\\.|[^\\']+)+')|''|(?>`(?>\\.|[^\\`]+)+`)|``))
UUID [A-Fa-f0-9]{8}-(?:[A-Fa-f0-9]{4}-){3}[A-Fa-f0-9]{12}

# Networking
MAC (?:%{CISCOMAC}|%{WINDOWSMAC}|%{COMMONMAC})
CISCOMAC (?:(?:[A-Fa-f0-9]{4}\.){2}[A-Fa-f0-9]{4})
WINDOWSMAC (?:(?:[A-Fa-f0-9]{2}-){5}[A-Fa-f0-9]{2})
COMMONMAC (?:(?:[A-Fa-f0-9]{2}:){5}[A-Fa-f0-9]{2})
IPV6 
((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:)))(%.+)?
IPV4