Re: [graylog2] debugging pipelines is... difficult
I added this Github issue so you can track the issue I mentioned in point number 2: https://github.com/Graylog2/graylog-plugin-pipeline-processor/issues/46 Cheers, Edmundo > On 18 Jul 2016, at 10:51, Edmundo Alvarez wrote: > > I spent some time debugging the issue, and I found two of them: > > 1. The when expression should be wrapped in a "to_bool" function, otherwise > the parser gets confused about it and replaces it with "false": > > to_bool(regex("[^0-9a-zA-Z]([0-9]+\\.[0-9]+\\.[0-9]+\\.[0-9]+)[^0-9a-zA-Z].*[^0-9a-zA-Z]([0-9]+\\.[0-9]+\\.[0-9]+\\.[0-9]+)[^0-9a-zA-Z]",to_string($message.message)).matches) > > 2. There seems to be some problems when handling strings containing > backslashes. You need to escape them so they get parsed, but then the escape > character is still being used in the regular expression. I will investigate > further and keep you posted on that. > > Cheers, > Edmundo > >> On 13 Jul 2016, at 12:31, Jason Haar wrote: >> >> >> On Mon, Jul 11, 2016 at 11:28 AM, Jason Haar wrote: >> If I take the regex I wrote in this rule (as per first email), replace '\\' >> with '\', then the regex works fine via egrep. It's a simple "when, do this" >> type statement: I can't see what's gone wrong in it >> >> Oh - and thanks to your comment about the regex needing to match the entire >> line, I put ".*" at the beginning and end - but it made no difference. Still >> no Cisco syslog messages (as above) match >> >> >> -- >> Cheers >> >> Jason Haar >> Information Security Manager, Trimble Navigation Ltd. >> Phone: +1 408 481 8171 >> PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Graylog Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to graylog2+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/graylog2/CAFChrgJZng%2Bzc-iZ%2Bv73%2Bd8Q6YatVATaDtj2R%3Dd7sR9iXZfbHQ%40mail.gmail.com. >> For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/BA27A691-42D6-46BD-80B5-988211F400B3%40graylog.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] debugging pipelines is... difficult
I spent some time debugging the issue, and I found two of them: 1. The when expression should be wrapped in a "to_bool" function, otherwise the parser gets confused about it and replaces it with "false": to_bool(regex("[^0-9a-zA-Z]([0-9]+\\.[0-9]+\\.[0-9]+\\.[0-9]+)[^0-9a-zA-Z].*[^0-9a-zA-Z]([0-9]+\\.[0-9]+\\.[0-9]+\\.[0-9]+)[^0-9a-zA-Z]",to_string($message.message)).matches) 2. There seems to be some problems when handling strings containing backslashes. You need to escape them so they get parsed, but then the escape character is still being used in the regular expression. I will investigate further and keep you posted on that. Cheers, Edmundo > On 13 Jul 2016, at 12:31, Jason Haar wrote: > > > On Mon, Jul 11, 2016 at 11:28 AM, Jason Haar wrote: > If I take the regex I wrote in this rule (as per first email), replace '\\' > with '\', then the regex works fine via egrep. It's a simple "when, do this" > type statement: I can't see what's gone wrong in it > > Oh - and thanks to your comment about the regex needing to match the entire > line, I put ".*" at the beginning and end - but it made no difference. Still > no Cisco syslog messages (as above) match > > > -- > Cheers > > Jason Haar > Information Security Manager, Trimble Navigation Ltd. > Phone: +1 408 481 8171 > PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 > > -- > You received this message because you are subscribed to the Google Groups > "Graylog Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to graylog2+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/graylog2/CAFChrgJZng%2Bzc-iZ%2Bv73%2Bd8Q6YatVATaDtj2R%3Dd7sR9iXZfbHQ%40mail.gmail.com. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/4A90E6BA-9C9C-4D9C-ADE8-787ADEFB1D54%40graylog.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] debugging pipelines is... difficult
On Mon, Jul 11, 2016 at 11:28 AM, Jason Haar wrote: > If I take the regex I wrote in this rule (as per first email), replace > '\\' with '\', then the regex works fine via egrep. It's a simple "when, do > this" type statement: I can't see what's gone wrong in it > Oh - and thanks to your comment about the regex needing to match the entire line, I put ".*" at the beginning and end - but it made no difference. Still no Cisco syslog messages (as above) match -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CAFChrgJZng%2Bzc-iZ%2Bv73%2Bd8Q6YatVATaDtj2R%3Dd7sR9iXZfbHQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] debugging pipelines is... difficult
On Fri, Jul 8, 2016 at 10:32 PM, Edmundo Alvarez wrote: > > It's hard to tell what is wrong from here, since we can't exactly see how > your messages look like. Could you share a couple of messages with us? > > Please be aware that at the moment, the "regex" function needs to match > the whole string: > https://github.com/Graylog2/graylog-plugin-pipeline-processor/issues/35 > Sure thing So that rule is attempting to extract any TWO ipv4 addresses detected in any form of syslog message. So to give you some examples of when that occurs, we'll look to Cisco firewalls/routers %ASA-4-106023: Deny tcp src inside:192.168.3.79/57577 dst outside: 54.171.242.51/843 by access-group "acl_inside" [0x2923dc37, 0x0] %ASA-7-710006: EIGRP request discarded from 192.168.23.1 to inside:224.0.0.10 %ASA-4-106023: Deny tcp src inside:192.168.4.52/62508 dst outside: 21.125.185.18/5287 by access-group If I take the regex I wrote in this rule (as per first email), replace '\\' with '\', then the regex works fine via egrep. It's a simple "when, do this" type statement: I can't see what's gone wrong in it I have another pipeline with two rules and it's working just fine - it seems to be the regex in this that is at fault, but I can't see how -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CAFChrgK8T6t_728ynFmH2ePHMx9dhsFYq4stfk1DVcyrdtCRPw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] debugging pipelines is... difficult
Hi Jason, It's hard to tell what is wrong from here, since we can't exactly see how your messages look like. Could you share a couple of messages with us? Please be aware that at the moment, the "regex" function needs to match the whole string: https://github.com/Graylog2/graylog-plugin-pipeline-processor/issues/35 We will release a Graylog 2.1.0 alpha really soon, and it would be really helpful to know what you think about the new pipeline simulator. It's still a work in progress, just as the pipeline processor, but hopefully will help you debug your pipelines and rules. Regards, Edmundo > On 07 Jul 2016, at 23:31, Jason Haar wrote: > > > On Wed, Jul 6, 2016 at 9:50 PM, Jochen Schalanda wrote: > there's something coming up in Graylog 2.1.0 which will vastly simplify > testing pipeline rules. > > That's great to hear. Any suggestions as to what's wrong with my rule? > > Thanks > > > -- > Cheers > > Jason Haar > Information Security Manager, Trimble Navigation Ltd. > Phone: +1 408 481 8171 > PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 > > -- > You received this message because you are subscribed to the Google Groups > "Graylog Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to graylog2+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/graylog2/CAFChrgL7rcbe_rFpciwxs%3D5%3Dh%3D%3DXC7E3mdXWyO-skSP4ZjidCg%40mail.gmail.com. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CE4236F5-A849-43F0-A8B2-BA9BD79A359C%40graylog.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] debugging pipelines is... difficult
Hi there First I want to say how wonderful the "extractor" webpage is: it's so easy to create AND TEST extractors... ...unfortunately the new pipelines (which I want to use as they are the official future) don't have the same testing capacity. Can someone tell me what's wrong with this rule: it should extract pairs of ipv4 addresses out of any message. The pipeline shows all messages flowing through it, but none "hit" this rule. Conversely, my existing extractor rule that does the same thing (but with different fieldnames) is triggering just fine - so this rule must be broken - but I lack the background in whatever Java-nightmare this is to debug it ;-) rule "function ExtractIPv4Pairs" when regex("[^0-9a-zA-Z]([0-9]+\\.[0-9]+\\.[0-9]+\\.[0-9]+)[^0-9a-zA-Z].*[^0-9a-zA-Z]([0-9]+\\.[0-9]+\\.[0-9]+\\.[0-9]+)[^0-9a-zA-Z]",to_string($message.message)).matches then let pair = regex("[^0-9a-zA-Z]([0-9]+\\.[0-9]+\\.[0-9]+\\.[0-9]+)[^0-9a-zA-Z].*[^0-9a-zA-Z]([0-9]+\\.[0-9]+\\.[0-9]+\\.[0-9]+)[^0-9a-zA-Z]",to_string($message.message),["src","dst"]); set_field("pipeSrcIPv4",to_ip(pair.src)); set_field("pipeDstIPv4",to_ip(pair.dst)); end Thanks PS: it would REALLY help if there were a bunch of sample rules that demonstrated the fundamentals. The one example really doesn't demonstrate enough -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +1 408 481 8171 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CAFChrg%2Ba7ijysDtX5MJAMqtmovLBMCgAZOZep6zNEHYX0h%2BQsw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.