Re: [rsyslog] liblognorm vs grok

Just created https://github.com/rsyslog/liblognorm/issues/236 El 20/12/16 a las 11:58, mosto...@gmail.com escribió: El 20/12/16 a las 11:55, Rainer Gerhards escribió: 2016-12-20 11:54 GMT+01:00 mostolog--- via rsyslog: Must first line be... "version=2" (v lowercase) this, seehttp://www.lib

Re: [rsyslog] liblognorm vs grok

2016-12-20 11:54 GMT+01:00 mostolog--- via rsyslog : > Must first line be... > > "version=2" (v lowercase) this, see http://www.liblognorm.com/files/manual/configuration.html#rulebase-versions Rainer > > or > > "Version=2" (V uppercase) > > ? > > El 14/12/16 a las 10:44, mosto...@gmail.com escr

Re: [rsyslog] liblognorm vs grok

Must first line be... "version=2" (v lowercase) or "Version=2" (V uppercase) ? El 14/12/16 a las 10:44, mosto...@gmail.com escribió: El 07/12/16 a las 21:00, Rainer Gerhards escribió: I'm getting /invalid field type 'alternative'/ when using it. Any ideas? rule=test:%[ {"type":"al

Re: [rsyslog] liblognorm vs grok

El 07/12/16 a las 21:00, Rainer Gerhards escribió: I'm getting /invalid field type 'alternative'/ when using it. Any ideas? rule=test:%[ {"type":"alternative","parser":[ {"type":"literal","text":"-"}, {"type":"word","name":"identd"} ]} ]% no idea Did you Set Version=2 i

Re: [rsyslog] liblognorm vs grok

Sent from phone, thus brief. Am 07.12.2016 20:10 schrieb "David Lang" : On Wed, 7 Dec 2016, mosto...@gmail.com wrote: you either use alternative or you have two different rule lines >> > I'm getting /invalid field type 'alternative'/ when using it. Any ideas? > > rule=test:%[ > {"type":"alte

Re: [rsyslog] liblognorm vs grok

On Wed, 7 Dec 2016, mosto...@gmail.com wrote: you either use alternative or you have two different rule lines I'm getting /invalid field type 'alternative'/ when using it. Any ideas? rule=test:%[ {"type":"alternative","parser":[ {"type":"literal","text":"-"}, {"type":"word","name":"ide

Re: [rsyslog] liblognorm vs grok

a literal space is always more efficent than whitespace, only use whitespace if there can be more than one space, or tabs Ok. just a note, the new syntax is not always better than the old syntax 127.0.0.1 - - [17/Mar/2016:18:15:06 +0100] "GET /redacted HTTP/1.1" 200 59506 type=@apache_com

Re: [rsyslog] liblognorm vs grok

On Wed, 7 Dec 2016, mosto...@gmail.com wrote: {"type":"@apache" name="."} ? actuall, %{"type":"@apache" name="."}% This is one of the places where I like to use the older, more compact syntax :-) Older/Compact doesn't seem to have an alternative, reason why I started using JSON syntax...rig

Re: [rsyslog] liblognorm vs grok

On Wed, 7 Dec 2016, mosto...@gmail.com wrote: I'm still trying to reproduce/understand what is happening and building a test case for the github issue if needed. Consider the following HTTP access lines: 127.0.0.1 - - [17/Mar/2016:18:15:06 +0100] "GET /redacted HTTP/1.1" 200 59506 127.0

Re: [rsyslog] liblognorm vs grok

that is the same type of bug, just for another type. just add a note that we need to allow end of line for all types, it's not limited to space. I'm missing code commenting...probably I'm going to switch back to ~doc tasks :P ___ rsyslog mailing l

Re: [rsyslog] liblognorm vs grok

{"type":"@apache" name="."} ? actuall, %{"type":"@apache" name="."}% This is one of the places where I like to use the older, more compact syntax :-) Older/Compact doesn't seem to have an alternative, reason why I started using JSON syntax...right?

Re: [rsyslog] liblognorm vs grok

I'm still trying to reproduce/understand what is happening and building a test case for the github issue if needed. Consider the following HTTP access lines: 127.0.0.1 - - [17/Mar/2016:18:15:06 +0100] "GET /redacted HTTP/1.1" 200 59506 127.0.0.1 - - [17/Mar/2016:18:15:24 +0100] "OPTION

Re: [rsyslog] liblognorm vs grok

On Wed, 7 Dec 2016, mosto...@gmail.com wrote: almost, %@apache% makes no more sense than %word%, you need to give the match a name so %log:@apache% would work, or if you want to move everything up a later (rather than having $!apache!ip) you could do %.:@apache% That should work How would t

Re: [rsyslog] liblognorm vs grok

On Wed, 7 Dec 2016, mosto...@gmail.com wrote: I think it's a problem, several of the types require a space at the end, and I think they should all be modified to allow either a space or a end-of-line. ack. It's on my list for early next year. better check if one exists, I also think David cre

Re: [rsyslog] liblognorm vs grok

On Wed, 7 Dec 2016, mosto...@gmail.com wrote: when troubleshooting things like this, create a rule file that is as minimal as you can get and parse with the -v option, it will show you what it's doing as it walks through the line. I don't see how it parsed each message. Perhaps a debug option

Re: [rsyslog] liblognorm vs grok

I think it's a problem, several of the types require a space at the end, and I think they should all be modified to allow either a space or a end-of-line. ack. It's on my list for early next year. better check if one exists, I also think David created one. This is for the liblognorm project.

Re: [rsyslog] liblognorm vs grok

when troubleshooting things like this, create a rule file that is as minimal as you can get and parse with the -v option, it will show you what it's doing as it walks through the line. I don't see how it parsed each message. Perhaps a debug option must be enabled? number of tree nodes:

Re: [rsyslog] liblognorm vs grok

almost, %@apache% makes no more sense than %word%, you need to give the match a name so %log:@apache% would work, or if you want to move everything up a later (rather than having $!apache!ip) you could do %.:@apache% That should work How would that be using JSON syntax? {"type":"@apache"

Re: [rsyslog] liblognorm vs grok

Should something like this work? {"type":"@apache"}, {"type":"alternative","parser":[ {}, { {"type":"whitespace"}, ... } ]} El 07/12/16 a las 11:08, Rainer Gerhards escribió: 2016-12-07 10:38 GMT+01:00 mosto...@gmail.com : In this case

Re: [rsyslog] liblognorm vs grok

2016-12-07 10:38 GMT+01:00 mosto...@gmail.com : > > In this case, I seem to remember that number is defined as being > followed > by a space, so you can't use it if the number is followed by a newline. I'll have to confirm that...but may I know why? Should I fill an issue

Re: [rsyslog] liblognorm vs grok

In this case, I seem to remember that number is defined as being followed by a space, so you can't use it if the number is followed by a newline. I'll have to confirm that...but may I know why? Should I fill an issue if it's indeed that way? I think it's a problem, several of the types requir

Re: [rsyslog] liblognorm vs grok

2016-12-07 9:11 GMT+01:00 David Lang : > On Wed, 7 Dec 2016, mosto...@gmail.com wrote: > >>> when troubleshooting things like this, create a rule file that is as >>> minimal as you can get and parse with the -v option, it will show you what >>> it's doing as it walks through the line. >> >> Ok :) >

Re: [rsyslog] liblognorm vs grok

On Wed, 7 Dec 2016, mosto...@gmail.com wrote: when troubleshooting things like this, create a rule file that is as minimal as you can get and parse with the -v option, it will show you what it's doing as it walks through the line. Ok :) In this case, I seem to remember that number is define

Re: [rsyslog] liblognorm vs grok

when troubleshooting things like this, create a rule file that is as minimal as you can get and parse with the -v option, it will show you what it's doing as it walks through the line. Ok :) In this case, I seem to remember that number is defined as being followed by a space, so you can't

Re: [rsyslog] liblognorm vs grok

On Mon, 5 Dec 2016, mosto...@gmail.com wrote: I forgot: With provided rule file...why I'm getting a bunch of this errors when using /usr/lib/lognorm/lognormalizer? { "originalmsg": "127.0.0.1 - - [17\/Mar\/2016:18:15:31 +0100] \"GET \/redacted\/page HTTP\/1.1\" 200 1234", "unparsed-data": "

Re: [rsyslog] liblognorm vs grok

On Mon, 5 Dec 2016, mosto...@gmail.com wrote: Hi Coming back to liblognorm, I have a few questions I'll love an expert reply. 0:D *- Documentation [1] states how to define a type, but not how to use it. Are we properly using defined type "apache" in the configuration below?* almost, %@ap

Re: [rsyslog] liblognorm vs grok

I forgot: With provided rule file...why I'm getting a bunch of this errors when using /usr/lib/lognorm/lognormalizer? { "originalmsg": "127.0.0.1 - - [17\/Mar\/2016:18:15:31 +0100] \"GET \/redacted\/page HTTP\/1.1\" 200 1234", "unparsed-data": "" } El 05/12/16 a las 15:41, mosto...@gmail.c

Re: [rsyslog] liblognorm vs grok

Hi Coming back to liblognorm, I have a few questions I'll love an expert reply. 0:D *- Documentation [1] states how to define a type, but not how to use it. Are we properly using defined type "apache" in the configuration below?* - Apache access log seem to have 2 formats: common and comb

Re: [rsyslog] liblognorm vs grok

Is that documentation stored on a github like rsyslog's? http://www.liblognorm.com/files/manual/index.html El 05/12/16 a las 11:15, David Lang escribió: On Mon, 5 Dec 2016, mosto...@gmail.com wrote: Hi. Is there an online liblognorm tester to check the rules we are writing? Otherwise, coul

Re: [rsyslog] liblognorm vs grok

On Mon, 5 Dec 2016, mosto...@gmail.com wrote: Hi. Is there an online liblognorm tester to check the rules we are writing? Otherwise, could you provide a testing guide (http://www.liblognorm.com/files/manual/installation.html#testing) to build lognormalizer to test? the liblognorm package i

Re: [rsyslog] liblognorm vs grok

Hi. Is there an online liblognorm tester to check the rules we are writing? Otherwise, could you provide a testing guide (http://www.liblognorm.com/files/manual/installation.html#testing) to build lognormalizer to test? El 04/10/16 a las 19:27, mosto...@gmail.com escribió: Hi Radu After

Re: [rsyslog] liblognorm vs grok

On Sat, 8 Oct 2016, Radu Gheorghe wrote: That's right, it's not so much about problems as conveniency/flexibility. For example, with grok.regex you can specify optional fields right in the middle of the pattern. With liblognorm/mmnormalize I have to repeat that rule with and without that field.

Re: [rsyslog] liblognorm vs grok

Getting some ideas from reading this. Thank you! On Tue, Oct 18, 2016 at 3:22 AM Radu Gheorghe wrote: > It look very very very very nice, Rainer! Thanks for publishing! > -- > Performance Monitoring * Log Analytics * Search Analytics > Solr & Elasticsearch Support * http://sematext.com/ > > > On

Re: [rsyslog] liblognorm vs grok

It took a while, but finally the thesis is online: https://www.fernuni-hagen.de/imperia/md/content/rechnerarchitektur/rainer_gerhards.pdf Rainer 2016-10-06 11:32 GMT+02:00 Rainer Gerhards : > 2016-10-06 11:23 GMT+02:00 mosto...@gmail.com : >> >> Totally agree...(actually, liblognorm is givi

Re: [rsyslog] liblognorm vs grok

Not speaking for Radu, buy I think he does not have problems, but Grok rules seem more convenient, and often that's really what they are. That's where custom types come in: if you have a good base set, then it really is not much difference in convenience. Unfortunately we don't have this yet. Rain

Re: [rsyslog] liblognorm vs grok

2016-10-06 11:23 GMT+02:00 mosto...@gmail.com : > > Totally agree...(actually, liblognorm is giving me segfaults :P) >>> >> I'll try to check next week when my current task is done. >> > I know you're busy...trying to contribute as much as I can with everything > I deal with on my daily work. > >

Re: [rsyslog] liblognorm vs grok

Totally agree...(actually, liblognorm is giving me segfaults :P) I'll try to check next week when my current task is done. I know you're busy...trying to contribute as much as I can with everything I deal with on my daily work. Liblognorm is based on work from my MSc Thesis. The thesis pap

Re: [rsyslog] liblognorm vs grok

2016-10-06 10:42 GMT+02:00 mosto...@gmail.com : > > > El 04/10/16 a las 20:31, Joe Blow escribió: > >> >> >> Regex should be avoided like the plague, at all costs. If you know your >> logs well enough to write a regex for them, why wouldn't you write a >> liblognorm rule instead? >> > Totally ag

Re: [rsyslog] liblognorm vs grok

El 04/10/16 a las 20:31, Joe Blow escribió: Regex should be avoided like the plague, at all costs. If you know your logs well enough to write a regex for them, why wouldn't you write a liblognorm rule instead? Totally agree...(actually, liblognorm is giving me segfaults :P) I use liblogno

Re: [rsyslog] liblognorm vs grok

Regex should be avoided like the plague, at all costs. If you know your logs well enough to write a regex for them, why wouldn't you write a liblognorm rule instead? I use liblognorm + rsyslog to forward to ES with very little overhead. If you like performance and scalability, use liblognorm.

[rsyslog] liblognorm vs grok

Hi Radu After reading http://lists.adiscon.net/pipermail/rsyslog/2013-December/035122.html and considering several years have passed, I would like to get some feedback of your experience, to help me choose between raw forwarding messages+logstash or split before forwarding with mmnormalize.

Re: [rsyslog] liblognorm vs grok

Thanks a lot, David! This clears up a lot of stuff. I'll start using mmnormalize then, and I'll bug you guys again if I bump into issues :) 2013/12/4 David Lang > On Wed, 4 Dec 2013, Radu Gheorghe wrote: > > Hi David, >> >> Thanks a lot for your reply! I will add my comments inline. >> >> 201

Re: [rsyslog] liblognorm vs grok

On Wed, 4 Dec 2013, Radu Gheorghe wrote: Hi David, Thanks a lot for your reply! I will add my comments inline. 2013/12/4 David Lang On Wed, 4 Dec 2013, Radu Gheorghe wrote: Hi list :) I'm trying to understand if mmnormalize is a good fit for parsing a high traffic of logs, given the fac

Re: [rsyslog] liblognorm vs grok

Hi David, Thanks a lot for your reply! I will add my comments inline. 2013/12/4 David Lang > On Wed, 4 Dec 2013, Radu Gheorghe wrote: > > Hi list :) >> >> I'm trying to understand if mmnormalize is a good fit for parsing a high >> traffic of logs, given the fact that events are really heteroge

Re: [rsyslog] liblognorm vs grok

On Wed, 4 Dec 2013, Radu Gheorghe wrote: Hi list :) I'm trying to understand if mmnormalize is a good fit for parsing a high traffic of logs, given the fact that events are really heterogeneous (think log4j logs, apache logs, whatever logs are commonly produced). My only frame of reference is

[rsyslog] liblognorm vs grok

Hi list :) I'm trying to understand if mmnormalize is a good fit for parsing a high traffic of logs, given the fact that events are really heterogeneous (think log4j logs, apache logs, whatever logs are commonly produced). My only frame of reference is Logstash's grok filter