Re: [graylog2] Re: grok extractors not working
No, this should not be a timezone issue, as the extractors are re-checked every second. Do you have a lot of Grok patterns maybe? On Saturday, 6 June 2015 00:43:46 UTC+2, Jesse Skrivseth wrote: > > I don't have much new to report other than the observation that it takes > *exactly* 2 hours for newly created extractors to take effect. Could this > be a clock/timezone issue? All servers are UTC with accurate clocks. What > is coming in from NXLog and the devices behind it I cannot guarantee, but I > can't think of a reason that would matter. > > On Monday, June 1, 2015 at 8:30:40 AM UTC-6, Jesse Skrivseth wrote: >> >> Thanks to everyone for continuing to pursue this odd issue. >> >> Arie - We are using nxlog-ce version 2.9.1347 >> >> Kay - I can't seem to recreate the problem (yet) in a test environment, >> whether 1.0.2 or 1.1.0. There are some (possibly irrelevant) differences >> between test and production, but I'll mention them anyway. Production is >> built on Amazon Web Services using the provided 1.0.2 AMIs. Test is running >> locally from the 1.0.2 OVA images in ESXi. Test and production have >> different volumes of data coming in and different devices are sending logs. >> The only place I seem to be having trouble is in production with messages >> coming from the one network appliance I am focused on at the moment. I >> haven't had a need to add/update extractors for other devices yet, so I'm >> not sure if the problem is limited to that one device or is universal for >> this production instance. I will test more generally, with more devices, >> and see if I can find a pattern. >> > -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: grok extractors not working
I don't have much new to report other than the observation that it takes *exactly* 2 hours for newly created extractors to take effect. Could this be a clock/timezone issue? All servers are UTC with accurate clocks. What is coming in from NXLog and the devices behind it I cannot guarantee, but I can't think of a reason that would matter. On Monday, June 1, 2015 at 8:30:40 AM UTC-6, Jesse Skrivseth wrote: > > Thanks to everyone for continuing to pursue this odd issue. > > Arie - We are using nxlog-ce version 2.9.1347 > > Kay - I can't seem to recreate the problem (yet) in a test environment, > whether 1.0.2 or 1.1.0. There are some (possibly irrelevant) differences > between test and production, but I'll mention them anyway. Production is > built on Amazon Web Services using the provided 1.0.2 AMIs. Test is running > locally from the 1.0.2 OVA images in ESXi. Test and production have > different volumes of data coming in and different devices are sending logs. > The only place I seem to be having trouble is in production with messages > coming from the one network appliance I am focused on at the moment. I > haven't had a need to add/update extractors for other devices yet, so I'm > not sure if the problem is limited to that one device or is universal for > this production instance. I will test more generally, with more devices, > and see if I can find a pattern. > -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: grok extractors not working
Thanks to everyone for continuing to pursue this odd issue. Arie - We are using nxlog-ce version 2.9.1347 Kay - I can't seem to recreate the problem (yet) in a test environment, whether 1.0.2 or 1.1.0. There are some (possibly irrelevant) differences between test and production, but I'll mention them anyway. Production is built on Amazon Web Services using the provided 1.0.2 AMIs. Test is running locally from the 1.0.2 OVA images in ESXi. Test and production have different volumes of data coming in and different devices are sending logs. The only place I seem to be having trouble is in production with messages coming from the one network appliance I am focused on at the moment. I haven't had a need to add/update extractors for other devices yet, so I'm not sure if the problem is limited to that one device or is universal for this production instance. I will test more generally, with more devices, and see if I can find a pattern. On Monday, June 1, 2015 at 4:33:08 AM UTC-6, Kay Roepke wrote: > > Jesse, > > We've just tried to reproduce this issue on 1.1.0-RC.1 but it works as > expected. > Could you give that a try in a test environment, please? > > Thanks, > Kay > -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: grok extractors not working
Jesse, We've just tried to reproduce this issue on 1.1.0-RC.1 but it works as expected. Could you give that a try in a test environment, please? Thanks, Kay On Saturday, 30 May 2015 13:42:52 UTC+2, Bernd Ahlers wrote: > > Jesse, > > thank you for the update. I created an issue in GitHub for this with a > link to this mailing list thread. > > https://github.com/Graylog2/graylog2-server/issues/1192 > > I also started to test with the detailed data you submitted but did not > see any problems. I was testing on 1.1.0-rc.1 though. > > Next step is to test all of this with 1.0.2 (which you are running). > > I will let you know once I have any updates. > > Thank you! > > Bernd > > Jesse Skrivseth [Fri, May 29, 2015 at 11:41:52AM -0700] wrote: > >I'm not sure why, but suddenly the extractors are working today without > any > >further action on my part. There seems to be a very long delay between > when > >an extractor is configured and when it is in effect, at least in this > >environment. > > > >Another thing to note is that the data on this input is TLS encrypted > GELF > >via TCP, and the data is coming in from NXLog using GELF_TCP. > > > >On Thursday, May 28, 2015 at 3:25:05 PM UTC-6, Kay Röpke wrote: > >> > >> I'm not an expert on the OVAs so I would recommend simply setting up a > >> test instance to check this. Or you can wait until I get to it in the > (my) > >> morning ;) > >> > >>> > >>> > > > >-- > >You received this message because you are subscribed to the Google Groups > "graylog2" group. > >To unsubscribe from this group and stop receiving emails from it, send an > email to graylog2+unsubscr...@googlegroups.com. > >For more options, visit https://groups.google.com/d/optout. > > > -- > Developer > > Tel.: +49 (0)40 609 452 077 > Fax.: +49 (0)40 609 452 078 > > TORCH GmbH - A Graylog company > Steckelhörn 11 > 20457 Hamburg > Germany > > Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 > Geschäftsführer: Lennart Koopmann (CEO) > -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: grok extractors not working
Hi, Are you using the latest version of NXLog? There was a problem in an older version concerning Graylog/GELF. Arie. Op vrijdag 29 mei 2015 20:41:52 UTC+2 schreef Jesse Skrivseth: > > I'm not sure why, but suddenly the extractors are working today without > any further action on my part. There seems to be a very long delay between > when an extractor is configured and when it is in effect, at least in this > environment. > > Another thing to note is that the data on this input is TLS encrypted GELF > via TCP, and the data is coming in from NXLog using GELF_TCP. > > On Thursday, May 28, 2015 at 3:25:05 PM UTC-6, Kay Röpke wrote: >> >> I'm not an expert on the OVAs so I would recommend simply setting up a >> test instance to check this. Or you can wait until I get to it in the (my) >> morning ;) >> >>> >>> -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: grok extractors not working
Jesse, thank you for the update. I created an issue in GitHub for this with a link to this mailing list thread. https://github.com/Graylog2/graylog2-server/issues/1192 I also started to test with the detailed data you submitted but did not see any problems. I was testing on 1.1.0-rc.1 though. Next step is to test all of this with 1.0.2 (which you are running). I will let you know once I have any updates. Thank you! Bernd Jesse Skrivseth [Fri, May 29, 2015 at 11:41:52AM -0700] wrote: >I'm not sure why, but suddenly the extractors are working today without any >further action on my part. There seems to be a very long delay between when >an extractor is configured and when it is in effect, at least in this >environment. > >Another thing to note is that the data on this input is TLS encrypted GELF >via TCP, and the data is coming in from NXLog using GELF_TCP. > >On Thursday, May 28, 2015 at 3:25:05 PM UTC-6, Kay Röpke wrote: >> >> I'm not an expert on the OVAs so I would recommend simply setting up a >> test instance to check this. Or you can wait until I get to it in the (my) >> morning ;) >> >>> >>> > >-- >You received this message because you are subscribed to the Google Groups >"graylog2" group. >To unsubscribe from this group and stop receiving emails from it, send an >email to graylog2+unsubscr...@googlegroups.com. >For more options, visit https://groups.google.com/d/optout. -- Developer Tel.: +49 (0)40 609 452 077 Fax.: +49 (0)40 609 452 078 TORCH GmbH - A Graylog company Steckelhörn 11 20457 Hamburg Germany Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 Geschäftsführer: Lennart Koopmann (CEO) -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: grok extractors not working
It should take roughly a second or two. Still we will investigate :) On May 29, 2015 8:41 PM, "Jesse Skrivseth" wrote: > I'm not sure why, but suddenly the extractors are working today without > any further action on my part. There seems to be a very long delay between > when an extractor is configured and when it is in effect, at least in this > environment. > > Another thing to note is that the data on this input is TLS encrypted GELF > via TCP, and the data is coming in from NXLog using GELF_TCP. > > On Thursday, May 28, 2015 at 3:25:05 PM UTC-6, Kay Röpke wrote: >> >> I'm not an expert on the OVAs so I would recommend simply setting up a >> test instance to check this. Or you can wait until I get to it in the (my) >> morning ;) >> >>> >>> -- > You received this message because you are subscribed to the Google Groups > "graylog2" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to graylog2+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: grok extractors not working
I'm not sure why, but suddenly the extractors are working today without any further action on my part. There seems to be a very long delay between when an extractor is configured and when it is in effect, at least in this environment. Another thing to note is that the data on this input is TLS encrypted GELF via TCP, and the data is coming in from NXLog using GELF_TCP. On Thursday, May 28, 2015 at 3:25:05 PM UTC-6, Kay Röpke wrote: > > I'm not an expert on the OVAs so I would recommend simply setting up a > test instance to check this. Or you can wait until I get to it in the (my) > morning ;) > >> >> -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: grok extractors not working
Much appreciated! On Thursday, May 28, 2015 at 3:25:05 PM UTC-6, Kay Röpke wrote: > > I'm not an expert on the OVAs so I would recommend simply setting up a > test instance to check this. Or you can wait until I get to it in the (my) > morning ;) > -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: grok extractors not working
I'm not an expert on the OVAs so I would recommend simply setting up a test instance to check this. Or you can wait until I get to it in the (my) morning ;) On May 28, 2015 11:23 PM, "Jesse Skrivseth" wrote: > I hear the upgrade path is still in the works, but is there a way to > upgrade in-place or at least without data loss? > > On Thursday, May 28, 2015 at 3:18:06 PM UTC-6, Kay Röpke wrote: >> >> Many thanks! >> >> I will have a look in the morning. >> In the meantime it would be helpful if you could give 1.1.0-beta.3 a >> shot. It ships with an updated java-grok library. >> You can find the link to an OVA for convenience at the end of this blog >> post: >> >> https://www.graylog.org/graylog-1-1-beta-3-is-now-available/ >> > -- > You received this message because you are subscribed to the Google Groups > "graylog2" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to graylog2+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: grok extractors not working
I hear the upgrade path is still in the works, but is there a way to upgrade in-place or at least without data loss? On Thursday, May 28, 2015 at 3:18:06 PM UTC-6, Kay Röpke wrote: > > Many thanks! > > I will have a look in the morning. > In the meantime it would be helpful if you could give 1.1.0-beta.3 a shot. > It ships with an updated java-grok library. > You can find the link to an OVA for convenience at the end of this blog > post: > > https://www.graylog.org/graylog-1-1-beta-3-is-now-available/ > -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Re: grok extractors not working
Could you please create an extractor that shows this behavior and export
its configuration?
If at all possible please include a couple of messages which should cause
extracted fields to show up.
Please also include all the necessary grok patterns.
Otherwise it is extremely difficult to reproduce and debug.
Which server version are you using?
Thanks!
On May 28, 2015 10:52 PM, "Jesse Skrivseth" wrote:
> Many hours later, I'm no closer to a solution. It seems to be completely
> unpredictable.
>
> I have a grok extractor named "XTM515_firewall". It looks like this:
>
> %{NOTSPACE:SerialNumber} %{SYSLOGPROG:MessageType}:
> msg_id=%{QUOTEDSTRING:MessageId} %{NOTSPACE:Action}
> %{NOTSPACE:SourceInterface} %{NOTSPACE:DestinationInterface}
> %{NOTSPACE:UNWANTED} %{WORD:Protocol} %{NOTSPACE:UNWANTED}
> %{NOTSPACE:UNWANTED} %{IP:SourceIP} %{IP:DestinationIP}
> %{NUMBER:SourcePort} %{NUMBER:DestinationPort} %{NOTPAREN:UNWANTED}
> \(%{NOTPAREN:RuleName}\)
>
> this looks complex, but it doesn't matter how simple or complex the grok
> is, the behavior is the same. If I create this now and apply it, none of
> the fields will be extracted even though the test against a message passed.
> If I delete (yes, delete entirely) the extractor, extracted fields may
> start showing up briefly. If I create a new extractor with the same name
> "XTM515_firewall", but I change "RuleName" to "NewRuleName", it may or may
> not begin extracting again, but it won't be named "NewRuleName" - it will
> still be processing the old definition of the rule.
>
> I'm at a loss at this point. :(
>
> --
> You received this message because you are subscribed to the Google Groups
> "graylog2" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to graylog2+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups
"graylog2" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: grok extractors not working
Many hours later, I'm no closer to a solution. It seems to be completely
unpredictable.
I have a grok extractor named "XTM515_firewall". It looks like this:
%{NOTSPACE:SerialNumber} %{SYSLOGPROG:MessageType}:
msg_id=%{QUOTEDSTRING:MessageId} %{NOTSPACE:Action}
%{NOTSPACE:SourceInterface} %{NOTSPACE:DestinationInterface}
%{NOTSPACE:UNWANTED} %{WORD:Protocol} %{NOTSPACE:UNWANTED}
%{NOTSPACE:UNWANTED} %{IP:SourceIP} %{IP:DestinationIP}
%{NUMBER:SourcePort} %{NUMBER:DestinationPort} %{NOTPAREN:UNWANTED}
\(%{NOTPAREN:RuleName}\)
this looks complex, but it doesn't matter how simple or complex the grok
is, the behavior is the same. If I create this now and apply it, none of
the fields will be extracted even though the test against a message passed.
If I delete (yes, delete entirely) the extractor, extracted fields may
start showing up briefly. If I create a new extractor with the same name
"XTM515_firewall", but I change "RuleName" to "NewRuleName", it may or may
not begin extracting again, but it won't be named "NewRuleName" - it will
still be processing the old definition of the rule.
I'm at a loss at this point. :(
--
You received this message because you are subscribed to the Google Groups
"graylog2" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: grok extractors not working
Something is wrong with my environment. I've deleted every extractor I have on all inputs, yet some of the previously defined extraction is still occurring as messages flow in. Newly created grok extractors don't work, nor do simple regex to extract a single term into a named field Very odd.. On Thursday, May 28, 2015 at 8:37:15 AM UTC-6, Jesse Skrivseth wrote: > > Jochen, > > After the extractor is created, I expected the fields to be available on > the message itself. I look at all messages in the last 5 minutes, visually > find a message that follows this structure, click on it to show the field > list, but none of the supposedly extracted fields show in the field list on > the right panel. Testing that exact message in the extractor does properly > show the fields that should be extracted. > > If I do a plain regex extractor, that does work. But it seems even the > simplest grok - find the first matching number and name it "number" doesn't > work. > > Maybe I don't understand how grok expressions must be formed. -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: grok extractors not working
Jochen, After the extractor is created, I expected the fields to be available on the message itself. I look at all messages in the last 5 minutes, visually find a message that follows this structure, click on it to show the field list, but none of the supposedly extracted fields show in the field list on the right panel. Testing that exact message in the extractor does properly show the fields that should be extracted. If I do a plain regex extractor, that does work. But it seems even the simplest grok - find the first matching number and name it "number" doesn't work. Maybe I don't understand how grok expressions must be formed. -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: grok extractors not working
Hi Jesse,
how exactly are you searching for those fields? Please be aware that
additional fields aren't analyzed and thus wildcard search (e. g.
"syslogprog:fire*") won't work.
Cheers,
Jochen
On Thursday, 28 May 2015 04:02:21 UTC+2, Jesse Skrivseth wrote:
>
> So I have a collection of Grok patterns, things like:
>
> ...
> # Syslog Dates: Month Day HH:MM:SS
> SYSLOGTIMESTAMP %{MONTH} +%{MONTHDAY} %{TIME}
> PROG (?:[\w._/%-]+)
> SYSLOGPROG %{PROG:program}(?:\[%{POSINT:pid}\])?
> SYSLOGHOST %{IPORHOST}
> SYSLOGFACILITY <%{NONNEGINT:facility}.%{NONNEGINT:priority}>
> HTTPDATE %{MONTHDAY}/%{MONTH}/%{YEAR}:%{TIME} %{INT}
> ...
>
>
> The grok patterns themselves don't matter, those work fine. What doesn't
> work is extracting these into fields. So I create an extractor like this:
>
> Type: grok
> Field: full_message
> Pattern: %{SYSLOGPROG:syslogprog}: msg_id=%{QUOTEDSTRING:msg_id}
> %{WORD:result}
>
> I test the pattern and get matches as I expect
>
> msg_id3000-0148programfirewallresultAllowsyslogprogfirewall
>
> I save the extractor and wait for messages to flow in. But those fields
> are never extracted when I search for them.
>
> I'm sure I'm omitting something obvious. Any ideas?
>
> Thanks!
>
--
You received this message because you are subscribed to the Google Groups
"graylog2" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
