[graylog2] Extractors with Wildcards Cause High CPU/Load Average
Hi all,
I've finally discovered the source of my excess CPU load and high load
averages on my Graylog nodes!
I've got a bunch of extractors that I use to pull information from my
vSphere platform's VMKernel logs.
The catch with these is that a lot of items in the message string vary
quite a bit, so finding a regex to match is quite difficult... read pretty
much impossible for my limited regex skills :)
The way I've worked around this is to use wildcards in the regex strings
and that seems to be causing my load average to go from ~0.4 to ~2 or even
more and the CPU's regularly peak at 100%.
Is this expected behaviour?
I recall an issue with earlier versions of Graylog where wildcards in
stream rules would cause this but I believe that was much improved in the
1.0 release and I have noticed that difference. I'm running 1.0.2 at
present.
Is there a similar improvement with extractors in 1.1 or is it being worked
on perhaps?
I intend to put 1.1 into my test lab early next week but it doesn't see
anywhere near as many messages/sec as Production so I won't really see any
indications until I get it into Production.
I've attached my current extractors.
Any feedback on this would be great, and in the meantime I'll start trying
to optimise my extractors a bit more to see if I can remove some wildcards.
Cheers, Pete
--
You received this message because you are subscribed to the Google Groups
graylog2 group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
{
extractors: [
{
condition_type: regex,
condition_value: (?i).*NMP: nmp_ThrottleLogForDevice.*,
converters: [],
cursor_strategy: copy,
extractor_config: {
regex_value: (?i).*NMP:.*Cmd (0x..).*
},
extractor_type: regex,
order: 0,
source_field: message,
target_field: Cmd,
title: ESXi: Extract SCSI Command
},
{
condition_type: regex,
condition_value: (?i).*NMP: nmp_ThrottleLogForDevice:.*,
converters: [],
cursor_strategy: copy,
extractor_config: {
regex_value: (?i).*NMP: nmp_ThrottleLogForDevice:.*dev \(.*?)\.*
},
extractor_type: regex,
order: 0,
source_field: message,
target_field: Device,
title: ESXi: Extract Device
},
{
condition_type: regex,
condition_value: (?i).*NMP: nmp_ThrottleLogForDevice:.*,
converters: [],
cursor_strategy: copy,
extractor_config: {
regex_value: (?i).*NMP: nmp_ThrottleLogForDevice:.*path \(.*?)\.*
},
extractor_type: regex,
order: 0,
source_field: message,
target_field: Path,
title: ESXi: Extract Path
},
{
condition_type: regex,
condition_value: (?i).*NMP: nmp_ThrottleLogForDevice.*,
converters: [],
cursor_strategy: copy,
extractor_config: {
regex_value: (?i).*NMP: nmp_ThrottleLogForDevice.*Failed:
(H:0x.{1,2} D:0x.{1,2} P:0x.{1,2}).*
},
extractor_type: regex,
order: 0,
source_field: message,
target_field: Status,
title: ESXi: Extract Host, Device, or NMP Plugin Status
},
{
condition_type: regex,
condition_value: (?i).*NMP: nmp_ThrottleLogForDevice,
converters: [],
cursor_strategy: copy,
extractor_config: {
regex_value: (?i).*NMP: nmp_ThrottleLogForDevice.*sense data:
(0x.{1,2} 0x.{1,2} 0x.{1,2}).*
},
extractor_type: regex,
order: 0,
source_field: message,
target_field: SenseData,
title: ESXi: Extract Sense Data
},
{
condition_type: regex,
condition_value: (?i).*NMP: nmp_ThrottleLogForDevice,
converters: [],
cursor_strategy: copy,
extractor_config: {
regex_value: (?i).*NMP: nmp_ThrottleLogForDevice.*Act:(.*)$
},
extractor_type: regex,
order: 0,
source_field: message,
target_field: Action,
title: ESXi: Extract pathing action
},
{
condition_type: regex,
condition_value: (?i).*Lost access to volume (.{8}-.{8}-.{4}-.{12})
\\(.+?\\)*,
converters: [],
cursor_strategy: copy,
extractor_config: {
regex_value: (?i).*Lost access to volume (.{8}-.{8}-.{4}-.{12})
\\(.+?\\)*
},
extractor_type: regex,
order: 0,
source_field: message,
target_field: VolumeID,
title: ESXi: Lost Access to Volume - Volume ID
},
{
condition_type: regex,
condition_value: (?i).*Lost access to volume (.{8}-.{8}-.{4}-.{12})
\\(.+?\\)*,
converters: [],
cursor_strategy: copy,
extractor_config: {
regex_value: (?i).*Lost access to volume .{8}-.{8}-.{4}-.{12}
\\((.*)\\) *
},
extractor_type: regex,
order: 0,
source_field: message,
target_field: Datastore,
title: ESXi: Lost Access to
[graylog2] Graylog 1.1 rpm update issue on 1 of 2
I yum updated both of my CentOS6 graylog servers to 1.1. The primary server where all the ES indexes reside seemed to have worked no problem. The second one that connects to the 1st seems to work perfectly in every way, BUT any attempt to Search results in the Oops message. I see no errors in the logs or the System Overview. Even my Dashboard with statistics on source message qty values works. -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Graphs in dashboard
Hi, That is not possible at the moment, sorry. Regards, Edmundo On 04 Jun 2015, at 22:25, Denis Fateyev de...@fateyev.com wrote: Hello there, Is there a way to show charts on a dashboard exactly as they seen preliminary in search results? In other words, I have this graph in search results: But the generated graph on the dashboard looks like this, with pie chart always on top: Thanks, --- wbr, Denis. -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] graylog OVA 1.1 upgrade
I am currently running an OVA version 1.0 I want to upgrade to 1.1 but don't know if it is a drop-in upgrade. there is a warning to verify that it is before proceeding as it will wipe out all your data. I don't want to wipe any data and cannot confirm or deny that it is or is not a drop-in. Thanks -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Are GROK-conversions implemented?
Ok, thats a semi-colon...
That works. This is awesome.
On Thursday, 4 June 2015 16:29:40 UTC+2, Kay Roepke wrote:
Hi!
Version 1.1.0 ships with data type conversions in Grok patterns, yes.
To verify I've just set one up, on a Raw TCP input.
It splits input like:
somekey:345345
into two fields: key and value.
https://lh3.googleusercontent.com/-6q_fygDhb8c/VXBgYkpaOaI/ADs/kPQmqNHP2mI/s1600/screenshot.png
The format for grok conversions is:
%{PATTERNAME:fieldname;dataype}
You cannot leave out the fieldname, so %{INT;int} won't work. (You want to
assign field names anyway for clarity).
I can then successfully get the field statistics on field 'value'.
HTH,
Kay
On Thursday, 4 June 2015 14:49:08 UTC+2, Martin René Mortensen wrote:
It is in the current 1.1 beta 3 ?
From the version notes:
*Server *
- Enhanced Grok support with type conversions (integers, doubles and
dates)
I can't get it to work thou - I add :int to a http status for example,
but the field just disappears.
On Thursday, 19 March 2015 17:29:15 UTC+1, Kay Röpke wrote:
Not in the version we are using right now, but we plan on upgrading the
java grok version to support this.
Best,
Kay
On Mar 19, 2015 3:01 PM, Jerri Son jerri...@gmail.com wrote:
Hi there,
reading this:
https://github.com/Graylog2/graylog2-server/issues/377#issuecomment-43436017
I was wondering
if GROK-conversion are implemented already?
The obvious uses are of course to convert string to ints/floats so we
can do math on logs.
As it is right now, when i add :int or :float the output field just
gets extended to ie myfieldname:int.
I might just have the syntax wrong though :)
Cheers!
--
You received this message because you are subscribed to the Google
Groups graylog2 group.
To unsubscribe from this group and stop receiving emails from it, send
an email to graylog2+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
graylog2 group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] graylog OVA 1.1 upgrade
We have currently one report that stream alerts got lost during the update and need to re-configured. So please backup your installation before, but the update is possible without bigger manual migration steps. On 4 June 2015 at 20:35, d...@wildcatracing.com wrote: I am currently running an OVA version 1.0 I want to upgrade to 1.1 but don't know if it is a drop-in upgrade. there is a warning to verify that it is before proceeding as it will wipe out all your data. I don't want to wipe any data and cannot confirm or deny that it is or is not a drop-in. Thanks -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- Developer Tel.: +49 (0)40 609 452 077 Fax.: +49 (0)40 609 452 078 TORCH GmbH - A Graylog Company Steckelhörn 11 20457 Hamburg Germany https://www.graylog.com https://www.torch.sh/ Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 Geschäftsführer: Lennart Koopmann (CEO) -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Graylog2 server High CPU, configuration questions
Thanks Henrik, I will keep this in mind when deciding. Might switch the instance out to the CPU optimised one with 4 vCPUs and see how that copes, then scale horizontally out after that. Thanks all for the input. On Wednesday, 3 June 2015 20:38:25 UTC+1, Henrik Johansen wrote: Hello Matt, The short answer is that Graylog eats CPU and lot’s of it … and heaps more than competing solutions given the same message rate. As an example, we’re seeing a performance of roughly 700 messages/sec per physical CPU core across our Graylog cluster(s) when keeping the utilisation across all 128+ CPU cores at ~75%. That’s purely for running our graylog-server nodes - ES runs on dedicated hardware. We do have a crazy amount of extractors and streams which certainly impact the numbers above and a crazy variation in terms of message size, etc so YMMV. Since you’re running on AWS I would not recommend using more than 4 vCPUS per instance - in fact I would recommend 2 vCPUS per instance and a suitable number of those for running multiple graylog-server nodes. There are so many factors involved when it comes to CPU scheduling in virtualised environments and unless you keep track of all the variables (host load, the hardware used, number of instances and their vCPU distribution, etc) you might get substantially lower performance when running with lots of vCPUS … — HenrikJ On 03 Jun 2015, at 16:19, Matt Hines matthi...@gmail.com javascript: wrote: Hi BKeep, The Elasticsearch node is fine, that's on a separate box. The Graylog JVM settings are default, as I looked at increasing them but when I looked most people said to keep it the default. I don't think it's a memory issue as the memory usage is minimal, even a load times. I'm sure it's due to the fact the Graylog node only has 2 vCPUs but wanted to confirm that really. The Memory usage has stayed below 50% the entire time. On Wednesday, 3 June 2015 14:40:30 UTC+1, BKeep wrote: I would add another vCPU core or two and some more RAM. Also, have you adjusted your JVM HEAP size for Elasticsearch and Graylog? I have three nodes setup for our logging cluster. The master node is Graylog, Mongod, and Elasticsearch setup as master only with a JVM HEAP of 4g for Graylog and 4g for Elasticsearch. The other two nodes are data storage only. Each of those has 4vCPU's and 12GB of RAM where Elasticsearch has HEAP of 6g. That configuration allows us max throughput of about 2500 logs per second give or take. I should also note these are VMware VM's running on B200 blade hardware (2.7Ghz Xeon cores, SAN storage, etc.) On 06/03/2015 03:21 AM, Matt Hines wrote: Hey Henrik, I'm not. That was the first thing that I found when looking into the issue. We have 1 input, a syslog UDP, but the force_rdns is unticked. Thanks, On Tuesday, 2 June 2015 18:33:14 UTC+1, Henrik Johansen wrote: Hello Matt, Are you by any chance running inputs that have the force_rdns flag set? --- HenrikJ On 2. jun. 2015 kl. 17.37.16 CEST, Matt Hines matthi...@gmail.com wrote: Hey guys, We've just updated our Graylog setup by moving the ES to another box to give it space to breathe. But now we're seeing a large increase in msg/s coming into the graylog-server, at peak times, between 600-1200. This amount is only going to get bigger as users increase. My issue is, when the number breaches the 600 msgs/s number, graylog starts to struggle and starts filling up its process buffer and the CPU ramps up to 100% and stays there until the peak period is over and it has cleared the buffer out. The box is an AWS box, 2CPU, 8GB RAM. I have no idea if this hardware should be enough for this number of messages? If so, is there some tweaking I can do to optimise the system? If not, what would you recommend, I know it's mainly CPU based but would it be best to cluster a second box, or increase this box? Also, at what point will I start to lose messages incoming? When the process buffer and journal are both full? We're running Graylog v1.0.2 Thanks for any help! -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+u...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+u...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+u...@googlegroups.com javascript:. For more options, visit
[graylog2] [ANNOUNCE] Graylog v1.1 GA has been released
Hey everybody, a quick heads up that we just released Graylog v1.1 GA: https://www.graylog.org/graylog-1-1-is-now-generally-available/ Hope you like it! Cheers, Lennart -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Clean installation exception
Hi, I just install Graylog on a Centos 6.6 machine, through the RPM. When i tried to start the server i'm getting the follow exception 2015-06-03T13:59:38.555-04:00 WARN [PluginLoader] Plugin directory /etc/rc.d/init.d/plugin does not exist, not loading plugins. 2015-06-03T13:59:38.557-04:00 INFO [CmdLineTool] Loaded plugins: [] 2015-06-03T13:59:38.586-04:00 ERROR [CmdLineTool] Invalid configuration com.github.joschi.jadconfig.ParameterException: Couldn't convert value tru to Boolean. at com.github.joschi.jadconfig.converters.BooleanConverter.convertFrom(BooleanConverter.java:25) at com.github.joschi.jadconfig.converters.BooleanConverter.convertFrom(BooleanConverter.java:11) at com.github.joschi.jadconfig.JadConfig.convertStringValue(JadConfig.java:160) at com.github.joschi.jadconfig.JadConfig.processClassFields(JadConfig.java:138) at com.github.joschi.jadconfig.JadConfig.process(JadConfig.java:99) at org.graylog2.bootstrap.CmdLineTool.readConfiguration(CmdLineTool.java:316) at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:161) at org.graylog2.bootstrap.Main.main(Main.java:58) Any idea how to fix it ? Regards -- Victor M. Gatica H. -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] graylog OVA 1.1 upgrade
Thanks I will give it a shot next week! I will update the post with my findings. On Thursday, June 4, 2015 at 1:55:27 PM UTC-6, Marius Sturm wrote: We have currently one report that stream alerts got lost during the update and need to re-configured. So please backup your installation before, but the update is possible without bigger manual migration steps. On 4 June 2015 at 20:35, da...@wildcatracing.com javascript: wrote: I am currently running an OVA version 1.0 I want to upgrade to 1.1 but don't know if it is a drop-in upgrade. there is a warning to verify that it is before proceeding as it will wipe out all your data. I don't want to wipe any data and cannot confirm or deny that it is or is not a drop-in. Thanks -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+u...@googlegroups.com javascript:. For more options, visit https://groups.google.com/d/optout. -- Developer Tel.: +49 (0)40 609 452 077 Fax.: +49 (0)40 609 452 078 TORCH GmbH - A Graylog Company Steckelhörn 11 20457 Hamburg Germany https://www.graylog.com https://www.torch.sh/ Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 Geschäftsführer: Lennart Koopmann (CEO) -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Graphs in dashboard
Hello there, Is there a way to show charts on a dashboard exactly as they seen preliminary in search results? In other words, I have this graph in search results: https://lh3.googleusercontent.com/--KiGiWvsEtg/VXCy5wjIPsI/GJo/jIsCGxcUMyc/s1600/glog01.png But the generated graph on the dashboard looks like this, with pie chart always on top: https://lh3.googleusercontent.com/-_jc2xJ5vytQ/VXCzOGIH2jI/GJw/e3_pAsJMJbg/s1600/glog02.png Thanks, --- wbr, Denis. -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Top list of common errors
I would like to have a dashboard widget which lists most common errors and how many they are. Is that possible? 3452 Really common error / exception 234 Some other error 12 Another Error 2 Error -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Are GROK-conversions implemented?
It is in the current 1.1 beta 3 ? From the version notes: *Server * - Enhanced Grok support with type conversions (integers, doubles and dates) I can't get it to work thou - I add :int to a http status for example, but the field just disappears. On Thursday, 19 March 2015 17:29:15 UTC+1, Kay Röpke wrote: Not in the version we are using right now, but we plan on upgrading the java grok version to support this. Best, Kay On Mar 19, 2015 3:01 PM, Jerri Son jerri...@gmail.com javascript: wrote: Hi there, reading this: https://github.com/Graylog2/graylog2-server/issues/377#issuecomment-43436017 I was wondering if GROK-conversion are implemented already? The obvious uses are of course to convert string to ints/floats so we can do math on logs. As it is right now, when i add :int or :float the output field just gets extended to ie myfieldname:int. I might just have the syntax wrong though :) Cheers! -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+u...@googlegroups.com javascript:. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups graylog2 group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
