[graylog2] Re: Graylog is increasing log's size

2016-09-13 Thread Daniel Kamiński
maybe you're indexing some unnecessary fields? try to disable less needed 
data, you can also strip them off with pipelines before they get processed 
further, also I heard that BTRFS with compression enabled works nice with 
ES,

W dniu poniedziałek, 12 września 2016 16:56:36 UTC+2 użytkownik Rômullo 
Furtado Beltrame napisał:
>
> Hey Guys, I have a question. Maybe you can help me.
>
> My Graylog 2 is increasing the size of the logs in 3x or more compared to 
> the other solutions or syslogs. I've already chose: best compress to 
> elasticsearch and optimized option.
>
> There's something that I can do to resolve this problem? My environment is 
> consuming more than 200GB/month  in 20mi of logs per day.
>
> If someone has an idea I would thanks so much. 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/d9ae71bc-84df-4fe3-99ab-9ef0b05a9d14%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


[graylog2] Re: Graylog is increasing log's size

2016-09-12 Thread Jochen Schalanda
Hi Rômullo,

Graylog (and Elasticsearch) are indexing all log messages you're sending to 
them and create many secondary data structures, such as a inverted index, 
for features like full-text search. These secondary data structures of 
course need a certain amount of disk space and memory.

If you merely want to store log messages as plain text files, you might 
need to resort to another log management solution.

Cheers,
Jochen

On Monday, 12 September 2016 16:56:36 UTC+2, Rômullo Furtado Beltrame wrote:
>
> Hey Guys, I have a question. Maybe you can help me.
>
> My Graylog 2 is increasing the size of the logs in 3x or more compared to 
> the other solutions or syslogs. I've already chose: best compress to 
> elasticsearch and optimized option.
>
> There's something that I can do to resolve this problem? My environment is 
> consuming more than 200GB/month  in 20mi of logs per day.
>
> If someone has an idea I would thanks so much. 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/d81668bc-c635-4e9e-b750-88b716225613%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.