Re: [rsyslog] mmnormalize with mutiple input: conditionals?
2016-12-02 8:44 GMT+01:00 David Lang : > On Fri, 2 Dec 2016, Rainer Gerhards wrote: > >> 2016-12-02 8:27 GMT+01:00 David Lang : >> These "possible sources" and "log type" and "multiple variables" are >> exactly what I would like to clarify. Depending on the fine details, >> there may be a simple solution, maybe not. At least something to think >> about. I am thinking about someting along the lines of e.g. imfile >> input with an extra metadata parameter, e.g. >> >> metadata=["!var=foo", "!bar=another text"] > > > while that would be very useful, it's not this situation. In this case he is > able to deal with this by setting tag to a structured value and then parse > it later rather than setting multiple metadata values. > > In this case, we're working much later in the process, where it doesn't > matter that the source was from an imfile. It's working with the data after > it's been parsed by mmnormalize and then cleaning up the things where > different logs still result in non-normalized data. ok, this wasn't clear to me. > > This could be because the source came in with names already (json, cef, > name-value, etc), or because you need to take multiple fields in the log and > combine them. > > if the liblognorm ruleset ammend=: line could assign variable contents, not > just constant strings, it would address 90% of the issues. can you give an example of what you think? I ask because liblognorm does not know rsyslog variables (it cannot, as it is not a rsyslog thingy). Rainer ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] omriemann configuration
For almost all of the parameters to the module, they _must_ vary by message. The only exceptions are things like TLS settings, or the remote host endpoint. Everything else is structured data about an event that happened elsewhere. Most fields can be omitted if there's no parameter set - it's unusual that we set a description on a metric for example. Really we only require host/metric/service - I think we should error if you try to send an event that doesn't contain these three fields at least. I'm absolutely happy with a json blob for setting custom fields; you're right to question their flexibility - they're just string key/value pairs appended to the end of the protobuf message, so a json blob is perfect. Thanks for the second opinion. I prefer the structured approach anyway. On Fri, 2 Dec 2016 at 07:50 David Lang wrote: > On Fri, 2 Dec 2016, Bob Gregory wrote: > > > Evening all, > > > > I've mostly finished my last personal project, so my thoughts are turning > > to omriemann. > > > > I'm trying to work out how we might configure the module. Riemann > requires > > that we send a protobuf encoded message containing a few pre-set fields, > > plus whatever additional fields we feel like forwarding. > > > > host: localhost > > service: cpu-load-average/1m > > state: ok > > time: 1480661786 > > description: "everything is perfectly fine" > > tags: ["laptop", "personal"] > > metric: 0.58 > > ttl: 120 > > my-custom-field: 27 > > > > This makes it unusual for an rsyslog module: usually rsyslog is happy to > > ship arbitrary strings to a destination and only cares about the > _framing_ > > of your data: omelasticsearch, ommysql, omkafka, omrelp etc. all accept > > some number of static parameters, plus a free-form template for the > actual > > message. > > > > Omriemann, in order to be useful, will need to impose some structure on > the > > message itself. > > > > How do people think we should configure the module so that people have > > flexibility over the host, metric value, metric name, and tags on a > > per-message basis? > > use a parameter to pass the variable name to use for the field, and have a > default if they aren't set. > > Also, think hard about the need to set them on a per-message basis. > > > I guess the simplest thing that could possibly work is defining a simple > > message format, eg. `host=foo; metric_f=0.6; > > service=rsyslog.impstats/utime; timestamp=1480661786` that messages need > to > > conform to. We can then parse out the key/value pairs in the module and > > encode them to protobuf. > > no, that way lies madness (I did something very similar in the first > iteration > of omudpspoof, but in my defense that was before we had the action() cal) > > > Alternatively, we could set up the structure of the message in the config > > itself, like this: > > > > action( > > type="omriemann" > > host="$hostname" > > metric="$!metric.value" > > service="$!metric.name") > > > > That seems more user-friendly, but rules out using custom fields. I guess > > I'd have to create a new template per-field during module begin. > > this is the right approach for the fixed fields. For defining custom > fields, can > you accept a JSON structure and do the right thing? > > Given that the protobuf needs to be pre-defined and exist on both sides, > how > much flexibility do you really have? > > > On a related note, I think I remember seeing some discussion of > conversion > > functions recently. Some of the fields need to valid integers, floats, > unix > > timestamps etc. What's the best way of parsing those out? > > you will be passed strings [1] and need to validate them (and figure out > what to > do if you are passed garbage) > > [1] timestamps are a possible exception to this. > > David Lang > ___ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] mmnormalize with mutiple input: conditionals?
On Fri, 2 Dec 2016, Rainer Gerhards wrote: This could be because the source came in with names already (json, cef, name-value, etc), or because you need to take multiple fields in the log and combine them. if the liblognorm ruleset ammend=: line could assign variable contents, not just constant strings, it would address 90% of the issues. can you give an example of what you think? I ask because liblognorm does not know rsyslog variables (it cannot, as it is not a rsyslog thingy). I'm just talking about referencing variables defined as a part of the rule that was just run. I've had cases where I parse two named items in a rule, but really will be wanting to treat them as a single item later, so being able to say a=$b+' '+$c would be very useful. or cases where the log arrives as json and I really want to combine fields in it, or rename them. I saw this a lot with windows logs, they like to have date and time as separate fields, or have names of fields that don't match up with the names used in other log sources, so a simple ammend=:a=$b would help a lot. There are cases where real logic/math is needed, and I think those are inappropriate for such things, but there's a lot that can be done with simple assignments that can contain variables. David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] omriemann configuration
I need to think a bit before casting a ballot, but a) json blob sounds great b) sounds useful for impstats -- impstats can generate json Raienr 2016-12-02 8:41 GMT+01:00 Bob Gregory : > Evening all, > > I've mostly finished my last personal project, so my thoughts are turning > to omriemann. > > I'm trying to work out how we might configure the module. Riemann requires > that we send a protobuf encoded message containing a few pre-set fields, > plus whatever additional fields we feel like forwarding. > > host: localhost > service: cpu-load-average/1m > state: ok > time: 1480661786 > description: "everything is perfectly fine" > tags: ["laptop", "personal"] > metric: 0.58 > ttl: 120 > my-custom-field: 27 > > This makes it unusual for an rsyslog module: usually rsyslog is happy to > ship arbitrary strings to a destination and only cares about the _framing_ > of your data: omelasticsearch, ommysql, omkafka, omrelp etc. all accept > some number of static parameters, plus a free-form template for the actual > message. > > Omriemann, in order to be useful, will need to impose some structure on the > message itself. > > How do people think we should configure the module so that people have > flexibility over the host, metric value, metric name, and tags on a > per-message basis? > > I guess the simplest thing that could possibly work is defining a simple > message format, eg. `host=foo; metric_f=0.6; > service=rsyslog.impstats/utime; timestamp=1480661786` that messages need to > conform to. We can then parse out the key/value pairs in the module and > encode them to protobuf. > > Alternatively, we could set up the structure of the message in the config > itself, like this: > > action( >type="omriemann" >host="$hostname" >metric="$!metric.value" >service="$!metric.name") > > That seems more user-friendly, but rules out using custom fields. I guess > I'd have to create a new template per-field during module begin. > > On a related note, I think I remember seeing some discussion of conversion > functions recently. Some of the fields need to valid integers, floats, unix > timestamps etc. What's the best way of parsing those out? > ___ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] omriemann configuration
The problem there is that I'd need to reformat the json output of impstats in order for it to fit this module. I might be tempted to add a separate output format to impstats for that case, though, because it seems perverse to make people do that templating themselves. We can amortize that work if we also support a statsd output, which seems like a logical next step. On Fri, 2 Dec 2016 at 08:12 Rainer Gerhards wrote: > I need to think a bit before casting a ballot, but > > a) json blob sounds great > b) sounds useful for impstats -- impstats can generate json > > Raienr > > 2016-12-02 8:41 GMT+01:00 Bob Gregory : > > Evening all, > > > > I've mostly finished my last personal project, so my thoughts are turning > > to omriemann. > > > > I'm trying to work out how we might configure the module. Riemann > requires > > that we send a protobuf encoded message containing a few pre-set fields, > > plus whatever additional fields we feel like forwarding. > > > > host: localhost > > service: cpu-load-average/1m > > state: ok > > time: 1480661786 > > description: "everything is perfectly fine" > > tags: ["laptop", "personal"] > > metric: 0.58 > > ttl: 120 > > my-custom-field: 27 > > > > This makes it unusual for an rsyslog module: usually rsyslog is happy to > > ship arbitrary strings to a destination and only cares about the > _framing_ > > of your data: omelasticsearch, ommysql, omkafka, omrelp etc. all accept > > some number of static parameters, plus a free-form template for the > actual > > message. > > > > Omriemann, in order to be useful, will need to impose some structure on > the > > message itself. > > > > How do people think we should configure the module so that people have > > flexibility over the host, metric value, metric name, and tags on a > > per-message basis? > > > > I guess the simplest thing that could possibly work is defining a simple > > message format, eg. `host=foo; metric_f=0.6; > > service=rsyslog.impstats/utime; timestamp=1480661786` that messages need > to > > conform to. We can then parse out the key/value pairs in the module and > > encode them to protobuf. > > > > Alternatively, we could set up the structure of the message in the config > > itself, like this: > > > > action( > >type="omriemann" > >host="$hostname" > >metric="$!metric.value" > >service="$!metric.name") > > > > That seems more user-friendly, but rules out using custom fields. I guess > > I'd have to create a new template per-field during module begin. > > > > On a related note, I think I remember seeing some discussion of > conversion > > functions recently. Some of the fields need to valid integers, floats, > unix > > timestamps etc. What's the best way of parsing those out? > > ___ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > ___ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] mmnormalize with mutiple input: conditionals?
2016-12-02 9:11 GMT+01:00 David Lang : > On Fri, 2 Dec 2016, Rainer Gerhards wrote: > >>> This could be because the source came in with names already (json, cef, >>> name-value, etc), or because you need to take multiple fields in the log >>> and >>> combine them. >>> >>> if the liblognorm ruleset ammend=: line could assign variable contents, >>> not >>> just constant strings, it would address 90% of the issues. >> >> >> can you give an example of what you think? I ask because liblognorm >> does not know rsyslog variables (it cannot, as it is not a rsyslog >> thingy). > > > I'm just talking about referencing variables defined as a part of the rule > that was just run. > > I've had cases where I parse two named items in a rule, but really will be > wanting to treat them as a single item later, so being able to say a=$b+' > '+$c would be very useful. > > or cases where the log arrives as json and I really want to combine fields > in it, or rename them. > > I saw this a lot with windows logs, they like to have date and time as > separate fields, or have names of fields that don't match up with the names > used in other log sources, so a simple ammend=:a=$b would help a lot. > > There are cases where real logic/math is needed, and I think those are > inappropriate for such things, but there's a lot that can be done with > simple assignments that can contain variables. Nothing I can do immediately, but can you pls open an issue tracker with some explanation (copy&paste?) on liblognorm. I have planned time early next year to work on enhancements and this would fit in. Rainer ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] omriemann configuration
2016-12-02 9:30 GMT+01:00 Bob Gregory : > The problem there is that I'd need to reformat the json output of impstats > in order for it to fit this module. I might be tempted to add a separate > output format to impstats for that case, though, because it seems perverse > to make people do that templating themselves. +1 for additonal format. Should work out of the box. Pls open issue tracker (and I don't mind if you create a PR for it). > > We can amortize that work if we also support a statsd output, which seems > like a logical next step. +1, also worth an issue Side-note: I like the issues as they can cleanly show what work is supposed to happen.I'll align all of them to the REK project in github. Rainer > > On Fri, 2 Dec 2016 at 08:12 Rainer Gerhards > wrote: > >> I need to think a bit before casting a ballot, but >> >> a) json blob sounds great >> b) sounds useful for impstats -- impstats can generate json >> >> Raienr >> >> 2016-12-02 8:41 GMT+01:00 Bob Gregory : >> > Evening all, >> > >> > I've mostly finished my last personal project, so my thoughts are turning >> > to omriemann. >> > >> > I'm trying to work out how we might configure the module. Riemann >> requires >> > that we send a protobuf encoded message containing a few pre-set fields, >> > plus whatever additional fields we feel like forwarding. >> > >> > host: localhost >> > service: cpu-load-average/1m >> > state: ok >> > time: 1480661786 >> > description: "everything is perfectly fine" >> > tags: ["laptop", "personal"] >> > metric: 0.58 >> > ttl: 120 >> > my-custom-field: 27 >> > >> > This makes it unusual for an rsyslog module: usually rsyslog is happy to >> > ship arbitrary strings to a destination and only cares about the >> _framing_ >> > of your data: omelasticsearch, ommysql, omkafka, omrelp etc. all accept >> > some number of static parameters, plus a free-form template for the >> actual >> > message. >> > >> > Omriemann, in order to be useful, will need to impose some structure on >> the >> > message itself. >> > >> > How do people think we should configure the module so that people have >> > flexibility over the host, metric value, metric name, and tags on a >> > per-message basis? >> > >> > I guess the simplest thing that could possibly work is defining a simple >> > message format, eg. `host=foo; metric_f=0.6; >> > service=rsyslog.impstats/utime; timestamp=1480661786` that messages need >> to >> > conform to. We can then parse out the key/value pairs in the module and >> > encode them to protobuf. >> > >> > Alternatively, we could set up the structure of the message in the config >> > itself, like this: >> > >> > action( >> >type="omriemann" >> >host="$hostname" >> >metric="$!metric.value" >> >service="$!metric.name") >> > >> > That seems more user-friendly, but rules out using custom fields. I guess >> > I'd have to create a new template per-field during module begin. >> > >> > On a related note, I think I remember seeing some discussion of >> conversion >> > functions recently. Some of the fields need to valid integers, floats, >> unix >> > timestamps etc. What's the best way of parsing those out? >> > ___ >> > rsyslog mailing list >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > http://www.rsyslog.com/professional-services/ >> > What's up with rsyslog? Follow https://twitter.com/rgerhards >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> ___ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> > ___ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Are we building an ERK stack?
El 01/12/16 a las 23:08, David Lang escribió: On Thu, 1 Dec 2016, mosto...@gmail.com wrote: I think that you are going to end up with some grief, if the message could not be insterted into ES for some reason, I think the odds are good that you will find that rawmsg can't be inserted either. After sending the email I though the same... I would keep the errorfile as a file and look at it periodially. I expect that when you first start things up, you will run into a number of errors, but once you work your way though them, the error rate will be low. Set your monitoring system to monitor the size of the errorfile, and it it starts growing significantly, generate an alert. Would love to have a more unattended/XXth century way, if anyone knows. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Are we building an ERK stack?
You may well be able to insert the rejected log into a different index. Most of our failed logs are down to a mismatch between the mapping config and the fields in json logs. An error index that treats the whole message as a single blob should work fine. On Fri, 2 Dec 2016, 08:35 mosto...@gmail.com, wrote: > El 01/12/16 a las 23:08, David Lang escribió: > > On Thu, 1 Dec 2016, mosto...@gmail.com wrote: > > > > I think that you are going to end up with some grief, if the message > > could not be insterted into ES for some reason, I think the odds are > > good that you will find that rawmsg can't be inserted either. > After sending the email I though the same... > > > I would keep the errorfile as a file and look at it periodially. I > > expect that when you first start things up, you will run into a number > > of errors, but once you work your way though them, the error rate will > > be low. > > > > Set your monitoring system to monitor the size of the errorfile, and > > it it starts growing significantly, generate an alert. > Would love to have a more unattended/XXth century way, if anyone knows. > ___ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Are we building an ERK stack?
On Fri, 2 Dec 2016, Bob Gregory wrote: You may well be able to insert the rejected log into a different index. Most of our failed logs are down to a mismatch between the mapping config and the fields in json logs. An error index that treats the whole message as a single blob should work fine. what bytes would need to be escaped? what if it's invalid unicode junk, etc. almost by definition we are talking about corrupt data. David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Are we building an ERK stack?
I'm not sure that's true in the general case. Of the errors I've had with our elk stack, upward of 95% have been caused by type errors (json field should be an int but is an object); some small handful have failed because a message was truncated somewhere asking the line; a smaller number have failed because somebody hand-crafted json and forgot about a trailing comma or quote. Overwhelmingly, the data aren't corrupted: they were invalid at source in a way that would still allow them to be read as plain Unicode strings. Obviously I accept that given enough data, I'll see more interesting failure modes that need more thought, but reading from the errorfile and pushing to a separate error index would work very well in our environment. On Fri, 2 Dec 2016, 08:43 David Lang, wrote: On Fri, 2 Dec 2016, Bob Gregory wrote: > You may well be able to insert the rejected log into a different index. > Most of our failed logs are down to a mismatch between the mapping config > and the fields in json logs. > > An error index that treats the whole message as a single blob should work > fine. what bytes would need to be escaped? what if it's invalid unicode junk, etc. almost by definition we are talking about corrupt data. David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] mmnormalize with mutiple input: conditionals?
Hi
Combined threads reply follow.
On Thu, 1 Dec 2016, David Lang wrote
sigh, this is getting a wee bit frustrating, you keep saying "it hurts
when I do X", we say "that doesn't work well, do Y" and you come back
a day or so later saying "but it really huts when I do X"... (it
doesn't help when we ask you to provide information and you instead
spend hours trying other things)
I understand you better than you think. I'm the pupil you never wanted.
Said so, in my defense, I would say I am testing the /Y/ things too.
now that I have expressed my frustration, you are finding bugs,
helping to fix them, and raising some good questions along the way.
Just understand why once in a while our answers seem a bit curt.
Don't worry. I understand it but I'm learning a lot along the way!. You
are having A LOT of patience and being really kind and very instructive.
while I can see the use cases for "call $.var", what would you do if
you call a ruleset that doesn't exist? you would first have to do 'if
$.var == [array of legal values] then' to be safe.
Sure, something like if $!var exists then { call $!var } (one if
statement vs many)
1. just a bunch of if statements
performance cost of doing a bunch of if tests
easy to include additional tests from a directory of files
That's my best option so far.
2/3. if then else if...
This saves up to 199 if's. Would this be noticiable?
4. switch statement
This seems interesting...
5. variable call statements
what to do if called ruleset doesn't exist?
I'm screwed
6. function lookup tables
This is the Nth time you mention those...start to think I *really* need
to start messing with them.
I think that with elsif, the need for switch (#4) is low, and the
restrictions of it only doing simple equivalence tests (no startswith,
contains, etc) really limit it's use
Totally agree.
call var (#5) seems easy to implement, but I really don't like opening
up the problem of calling a non-existant ruleset. We could have it
silently do nothing, but that gets really messy and I am already
cringing at the troubleshooting exhanges we will have to help people
figure out what is/isn't happeing.
Is there a *exists* statement? (eg: /if exists "object-name" then/)
function pointers are by far the most complicated, and since they
include ruleset parsing after startup, they have the potential to be
really ugly to implement. On the other hand, they are also by far the
most powerful. If we could do things like limiting the functions so
that they can't do any of the startup-type things[1] and only include
statements that are normally executed for each log type, this would
also give us a back-door way of providing the dynamic configuration
that many people have been asking for.
"Limiting the functions" sounds too /patchy/ for me.
On Thu, 1 Dec 2016, David Lang wrote
if then {
set common things
}
$includeconfig /etc/rsyslog.d/apps.d/*.conf
else {
unknown app
}
in each of the apps.d/*.conf files do
else if then {
stuff
}
This is what I had in mind.
(as an optimization, make the most common apps the earliest in the
directory)
Loved alphabetical order, but now I understand why this is better.
On Fri, 2 Dec 2016, Rainer Gerhards wrote
What exactly do you do with the variables you set inside the if body?
Are they always the same? Where does the data originate from?
I try to understand the scenario better, because I vaguely think I may
be able to find a much simpler solution which would require possible
minimal code changes. But I can't express myself clearer at the
moment, it's more a gut feeling.
It would be good if you could post some *concrete* example of three or
four of the if blocks, that would definitely aid understanding.
As David said, I would love having separated config files for each app to:
1. normalize
2. define this application pipeline (after normalization, invoke geoip,
add some fields to JSON...)
3. index or drop messages at app criterion.
Main config would be some common routines (like index)+combined app.conf
files.
On Fri, 2 Dec 2016, David Lang wrote
if the liblognorm ruleset ammend=: line could assign variable
contents, not just constant strings, it would address 90% of the issues.
It would let me add/remove/edit fields. Other things, like geoip
tagging, should still be made manually (I guess)
On Fri, 2 Dec 2016, Rainer Gerhards wrote
Nothing I can do immediately, but can you pls open an issue tracker
with some explanation (copy&paste?) on liblognorm. I have planned time
early next year to work on enhancements and this would fit in.
Music to my ears
Thank you both a lot for you replies.
Not only learning a lot, but feeling in debt with you, reason why I'm
contributing as much as I can in the most ways I'm able.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
Re: [rsyslog] Are we building an ERK stack?
On Fri, 2 Dec 2016, Bob Gregory wrote: I'm not sure that's true in the general case. Of the errors I've had with our elk stack, upward of 95% have been caused by type errors (json field should be an int but is an object); some small handful have failed because a message was truncated somewhere asking the line; a smaller number have failed because somebody hand-crafted json and forgot about a trailing comma or quote. Overwhelmingly, the data aren't corrupted: they were invalid at source in a way that would still allow them to be read as plain Unicode strings. Obviously I accept that given enough data, I'll see more interesting failure modes that need more thought, but reading from the errorfile and pushing to a separate error index would work very well in our environment. I get _really_ nervous about even low probability failure modes in my failure paths. Murphy likes me too much :-) doing it your way, you still have the failedlog messages from your failure path that you will need to monitor, so you have reduced the scope of the problem, but still have the same basic problem. David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] mmnormalize with mutiple input: conditionals?
(as an optimization, make the most common apps the earliest in the directory) And that's where that /if exists call $!tag/ idea makes more sense, cause you don't need to compare, neither to arrange most used applications at top. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] mmnormalize with mutiple input: conditionals?
On Fri, 2 Dec 2016, mosto...@gmail.com wrote:
while I can see the use cases for "call $.var", what would you do if you
call a ruleset that doesn't exist? you would first have to do 'if $.var ==
[array of legal values] then' to be safe.
Sure, something like if $!var exists then { call $!var } (one if statement vs
many)
remember that values in logs may be under the control of an attacker (just about
every field can be hand-crafted by a person sending you a log message). I know
you are thinking of the simple case where you are only reading files and
processing them, but as soon as you support logs from any other sources
(including writing logs to /dev/log), things get much messier.
some way to test if a ruleset by that name exists would be needed, but keep in
mind that you may have other rulesets that exist besides the ones you want to
call.
1. just a bunch of if statements
performance cost of doing a bunch of if tests
easy to include additional tests from a directory of files
That's my best option so far.
2/3. if then else if...
This saves up to 199 if's. Would this be noticiable?
it depends on the volume of logs, but yes, it can be noticable, especially if
you are in the situation of most people where a few log sources generate a large
percentage of your logs.
note that you can use stop in a traditional if statement to avoid wasting time
processing any other rules after that point
4. switch statement
This seems interesting...
but note it's limited to exact matches, there's a lot of times where
'startswith' is really a good fit. In your case, the same app run by multiple
teams, or multiple instances of the same app (and therefor the same log
manipulation being needed) could be combined with the other options, but not
with a switch.
In it's basic implementation, a switch is just a fancy way of typing if then
else statements.
With what Rainer pointed out about
if then
else if
else if
else
I don't see a switch being noticably better.
5. variable call statements
what to do if called ruleset doesn't exist?
I'm screwed
6. function lookup tables
This is the Nth time you mention those...start to think I *really* need to
start messing with them.
they are an incredibly powerful tool to deal with distilling a lot of stuff down
to a few results.
geoip lookups were one of the use cases I was thinking of when I created the
spec for lookup tables, but another was categorization, either giving it a list
of machine and what environment they are in (dev/QA/prod/DR/etc) so that alerts
could be adjusted/redirected/silenced depending on what system generated the
identical log message.
the problem of calling a non-existant ruleset. We could have it silently do
nothing, but that gets really messy and I am already cringing at the
troubleshooting exhanges we will have to help people figure out what
is/isn't happeing.
Is there a *exists* statement? (eg: /if exists "object-name" then/)
not currently (there is an open issue for it)
and an exists statement would (at least initially) just be checking for a
variable existing (as opposed to if $.var = '' that we have to do now)
checking if a ruleset exists is a very different thing,
function pointers are by far the most complicated, and since they include
ruleset parsing after startup, they have the potential to be really ugly to
implement. On the other hand, they are also by far the most powerful. If we
could do things like limiting the functions so that they can't do any of
the startup-type things[1] and only include statements that are normally
executed for each log type, this would also give us a back-door way of
providing the dynamic configuration that many people have been asking for.
"Limiting the functions" sounds too /patchy/ for me.
The problem is that there are a lot of things that only really make sense at
startup. A lot of them have no way to undo them later.
loading modules is a good case in point. loading a module not only adds code, it
adds syntax, there isn't any sane way to unload a module, so if a function
loaded a module and it was redefined to not load the module, behavior would be
different than if the module was never loaded in the first place.
another problem is that a lot of things get defined before the multiple threads
are forked, and they then go on their separate ways. Trying to redefine these
things across all threads in a consistant manner would be a nightmare.
and if you think about what you want to do differently for different logs, when
would you need to change some global state? 99.9%+ of the time, you want to
change variable assignements, do if statements, change where the output goes
adding all that complexity and risk to support the <0.1% use case would be
enough to eliminate any possibility of it being implemented (even if Rainer
thought it was a good idea, the work needed to implement it would delay it
forever)
going for the limited version
Re: [rsyslog] mmnormalize with mutiple input: conditionals?
2016-12-02 10:44 GMT+01:00 mosto...@gmail.com : > >> (as an optimization, make the most common apps the earliest in the >> directory) > > And that's where that /if exists call $!tag/ idea makes more sense, cause > you don't need to compare, neither to arrange most used applications at top. I, too, think this idea is useful. We just need to make sure that a good error message is emitted when the ruleset cannot be found. It's conceptually a function pointer in C. The problem is that many people throw away rsyslog messages, it's even default in most distros. That's also the reason why I change the error message system so that at least on systemd systems the error messages will go into the systemctl status display. I hope this helps resolving all these unnecessary grief caused by ignored error messages. Rainer ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] mmnormalize with mutiple input: conditionals?
2016-12-02 10:50 GMT+01:00 David Lang :
> On Fri, 2 Dec 2016, mosto...@gmail.com wrote:
>
>>> while I can see the use cases for "call $.var", what would you do if you
>>> call a ruleset that doesn't exist? you would first have to do 'if $.var ==
>>> [array of legal values] then' to be safe.
>>
>> Sure, something like if $!var exists then { call $!var } (one if statement
>> vs many)
>
>
> remember that values in logs may be under the control of an attacker (just
> about every field can be hand-crafted by a person sending you a log
> message). I know you are thinking of the simple case where you are only
> reading files and processing them, but as soon as you support logs from any
> other sources (including writing logs to /dev/log), things get much messier.
>
> some way to test if a ruleset by that name exists would be needed, but keep
> in mind that you may have other rulesets that exist besides the ones you
> want to call.
>
>>> 1. just a bunch of if statements
>>>
>>> performance cost of doing a bunch of if tests
>>> easy to include additional tests from a directory of files
>>
>> That's my best option so far.
>>
>>> 2/3. if then else if...
>>
>> This saves up to 199 if's. Would this be noticiable?
>
>
> it depends on the volume of logs, but yes, it can be noticable, especially
> if you are in the situation of most people where a few log sources generate
> a large percentage of your logs.
>
> note that you can use stop in a traditional if statement to avoid wasting
> time processing any other rules after that point
in an else if chain, the chain evaluation stops when the first
condition is true. That's because then the "then" part becomes
executed, and the "else" not. The "else" part is the rest of the chain
(think grammar given yesterday).
>
>>> 4. switch statement
>>
>> This seems interesting...
>
>
> but note it's limited to exact matches, there's a lot of times where
> 'startswith' is really a good fit. In your case, the same app run by
> multiple teams, or multiple instances of the same app (and therefor the same
> log manipulation being needed) could be combined with the other options, but
> not with a switch.
>
> In it's basic implementation, a switch is just a fancy way of typing if then
> else statements.
switch is primarily a tool for compiler optimization IMHO and makes
sense when scalar values are evaluated. This can then be replaced by
jump tables, reducing O(n) evaluation time to O(1). It doesn't gain
you anything if used on vector values (like strings).
Other than that, switch is just another way of wrinting "else if" chains.
>
> With what Rainer pointed out about
>
> if then
> else if
> else if
> else
>
> I don't see a switch being noticably better.
lol, yup. totally equivalent.
>
>>> 5. variable call statements
>>>
>>> what to do if called ruleset doesn't exist?
>>
>> I'm screwed
>>
>>> 6. function lookup tables
>>
>> This is the Nth time you mention those...start to think I *really* need to
>> start messing with them.
>
>
> they are an incredibly powerful tool to deal with distilling a lot of stuff
> down to a few results.
>
> geoip lookups were one of the use cases I was thinking of when I created the
> spec for lookup tables, but another was categorization, either giving it a
> list of machine and what environment they are in (dev/QA/prod/DR/etc) so
> that alerts could be adjusted/redirected/silenced depending on what system
> generated the identical log message.
>
>>> the problem of calling a non-existant ruleset. We could have it silently
>>> do nothing, but that gets really messy and I am already cringing at the
>>> troubleshooting exhanges we will have to help people figure out what
>>> is/isn't happeing.
>>
>> Is there a *exists* statement? (eg: /if exists "object-name" then/)
>
>
> not currently (there is an open issue for it)
>
> and an exists statement would (at least initially) just be checking for a
> variable existing (as opposed to if $.var = '' that we have to do now)
>
> checking if a ruleset exists is a very different thing,
if I have the name in the compiled config, it's easy to implement. If
not, it's some more work. Need to check. But my focus is currently not
on this, can be a side-activity.
Rainer
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
Re: [rsyslog] mmnormalize with mutiple input: conditionals?
important word missing: switch is primarily a tool for compiler optimization IMHO and **only** makes sense when scalar values are evaluated. This can then be replaced by jump tables, reducing O(n) evaluation time to O(1). It doesn't gain you anything if used on vector values (like strings). Other than that, switch is just another way of wrinting "else if" chains. Sorry for that, Rainer ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Are we building an ERK stack?
2016-12-02 10:31 GMT+01:00 David Lang : > On Fri, 2 Dec 2016, Bob Gregory wrote: > >> I'm not sure that's true in the general case. >> >> Of the errors I've had with our elk stack, upward of 95% have been caused >> by type errors (json field should be an int but is an object); some small >> handful have failed because a message was truncated somewhere asking the >> line; a smaller number have failed because somebody hand-crafted json and >> forgot about a trailing comma or quote. >> Overwhelmingly, the data aren't corrupted: they were invalid at source in >> a >> way that would still allow them to be read as plain Unicode strings. >> >> Obviously I accept that given enough data, I'll see more interesting >> failure modes that need more thought, but reading from the errorfile and >> pushing to a separate error index would work very well in our environment. > > > I get _really_ nervous about even low probability failure modes in my > failure paths. Murphy likes me too much :-) > > doing it your way, you still have the failedlog messages from your failure > path that you will need to monitor, so you have reduced the scope of the > problem, but still have the same basic problem. FYI: the original intent of the error file was to provide errors in a way that makes it easy to (semi?) automatically handle them via a different procedure (which my re-inject them once the problem has been solved). Rainer ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] mmnormalize with mutiple input: conditionals?
El 02/12/16 a las 10:50, David Lang escribió: remember that values in logs may be under the control of an attacker (just about every field can be hand-crafted by a person sending you a log message). I know you are thinking of the simple case where you are only reading files and processing them, but as soon as you support logs from any other sources (including writing logs to /dev/log), things get much messier. some way to test if a ruleset by that name exists would be needed, but keep in mind that you may have other rulesets that exist besides the ones you want to call. Didn't think about that. For sure that's an exploit vector. I don't see a switch being noticably better. Agree ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] mmnormalize with mutiple input: conditionals?
2016-12-02 11:45 GMT+01:00 mosto...@gmail.com : > El 02/12/16 a las 10:50, David Lang escribió: >> >> remember that values in logs may be under the control of an attacker (just >> about every field can be hand-crafted by a person sending you a log >> message). I know you are thinking of the simple case where you are only >> reading files and processing them, but as soon as you support logs from any >> other sources (including writing logs to /dev/log), things get much messier. >> >> some way to test if a ruleset by that name exists would be needed, but >> keep in mind that you may have other rulesets that exist besides the ones >> you want to call. > > Didn't think about that. For sure that's an exploit vector. to some extend you could mitigate this by call "prefix-" & $syslogtag just FYI (if we permit an expression at all, we permit full expresibility). Rainer ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
[rsyslog] REK stack
Hi all, I start a new thread as the other one has a million of different topics now ;-) Just a short note: I think we should finally call this projekt "REK stack" vs. ERK and other ideas. This seems to be consensus, is logical (rsyslog-ES-Kibna, in right order) and as Brian pointed out there already is prior art ;-). Violent objections please here. I have updated the rsyslog github REK project: https://github.com/rsyslog/rsyslog/projects/1 Rainer ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] mmnormalize with mutiple input: conditionals?
Then, implementing (although seems hard and you already said not intended) something like /if exists $!rulesetname then/ could improve if/else performance, cause using direct pointers. Anyhow I'm moving to existent/working if/else solution for short-term El 02/12/16 a las 11:48, Rainer Gerhards escribió: 2016-12-02 11:45 GMT+01:00 mosto...@gmail.com : El 02/12/16 a las 10:50, David Lang escribió: remember that values in logs may be under the control of an attacker (just about every field can be hand-crafted by a person sending you a log message). I know you are thinking of the simple case where you are only reading files and processing them, but as soon as you support logs from any other sources (including writing logs to /dev/log), things get much messier. some way to test if a ruleset by that name exists would be needed, but keep in mind that you may have other rulesets that exist besides the ones you want to call. Didn't think about that. For sure that's an exploit vector. to some extend you could mitigate this by call "prefix-" & $syslogtag just FYI (if we permit an expression at all, we permit full expresibility). Rainer ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] REK stack
Big +1, because "erk" sounds like the noise you make when somebody stands on your toe at a formal social event; Rek Project makes us sound like dangerous anarchists, or possibly a dub-techno outfit. On Fri, 2 Dec 2016 at 10:49 Rainer Gerhards wrote: Hi all, I start a new thread as the other one has a million of different topics now ;-) Just a short note: I think we should finally call this projekt "REK stack" vs. ERK and other ideas. This seems to be consensus, is logical (rsyslog-ES-Kibna, in right order) and as Brian pointed out there already is prior art ;-). Violent objections please here. I have updated the rsyslog github REK project: https://github.com/rsyslog/rsyslog/projects/1 Rainer ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] REK stack
TREK (long journey) seems a more friendly name for me. It also stands for "(The) Rsyslog ElasticSearch Kibana" project/stack :P (I can't stop posting on this list) El 02/12/16 a las 11:48, Rainer Gerhards escribió: Hi all, I start a new thread as the other one has a million of different topics now ;-) Just a short note: I think we should finally call this projekt "REK stack" vs. ERK and other ideas. This seems to be consensus, is logical (rsyslog-ES-Kibna, in right order) and as Brian pointed out there already is prior art ;-). Violent objections please here. I have updated the rsyslog github REK project: https://github.com/rsyslog/rsyslog/projects/1 Rainer ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] REK stack
Rek Project makes us sound like dangerous anarchists, or possibly a dub-techno outfit. LOL ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] REK stack
Hahaha. I agree. "erk" sounds like the noise I make when my elasticsearch cluster goes red on friday night when I was just headed out the door. ;) On Fri, Dec 2, 2016 at 5:57 AM Bob Gregory wrote: > Big +1, because "erk" sounds like the noise you make when somebody stands > on your toe at a formal social event; Rek Project makes us sound like > dangerous anarchists, or possibly a dub-techno outfit. > > On Fri, 2 Dec 2016 at 10:49 Rainer Gerhards > wrote: > > Hi all, > > I start a new thread as the other one has a million of different topics now > ;-) > > Just a short note: I think we should finally call this projekt "REK > stack" vs. ERK and other ideas. This seems to be consensus, is logical > (rsyslog-ES-Kibna, in right order) and as Brian pointed out there > already is prior art ;-). > > Violent objections please here. I have updated the rsyslog github REK > project: > > https://github.com/rsyslog/rsyslog/projects/1 > > Rainer > ___ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. > ___ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] filters question
Okay... I've made some changes to my configs using the output from the debug.
I'm now using "fromhost_ip ==" and statically listing every possible IP in the
array, and still some messages are falling through to my Unclassified.
Probably better to show than to try and explain...
>From /etc/rsyslogd.conf:
### for debug
*.* /var/splunk-syslog/msgdebug.log;RSYSLOG_DebugFormat
>From debug file:
Debug line with all properties:
FROMHOST: 'sdcubpe08.mycompany.com', fromhost-ip: '100.20.20.218', HOSTNAME:
'sdcubpe08.mycompany.com', PRI: 167,
syslogtag 'Vpxa:', programname: 'Vpxa', APP-NAME: 'Vpxa', PROCID: '-', MSGID:
'-',
TIMESTAMP: 'Dec 2 14:20:09', STRUCTURED-DATA: '-',
msg: ' verbose vpxa[2ACE2B70] [Originator@6876 sub=halservices
opID=WFU-32dbe2e3] [VpxaHalServices] HostChanged Event Fired, properties
changed [runtime.healthSystemRuntime]'
escaped msg: ' verbose vpxa[2ACE2B70] [Originator@6876 sub=halservices
opID=WFU-32dbe2e3] [VpxaHalServices] HostChanged Event Fired, properties
changed [runtime.healthSystemRuntime]'
inputname: imtcp rawmsg: '<167>2016-12-02T14:20:09.131Z sdcubpe08.mycompany.com
Vpxa: verbose vpxa[2ACE2B70] [Originator@6876 sub=halservices
opID=WFU-32dbe2e3] [VpxaHalServices] HostChanged Event Fired, properties
changed [runtime.healthSystemRuntime]'
$!:
$.:
$/:
>From my /etc/rsyslog.d/ESXi.conf
template(name="ESXi_app" type="string"
string="/var/splunk-syslog/ESXi/%FROMHOST%/%FROMHOST%-%$NOW%.log")
if $fromhost-ip == ["100.31.20.101",
"100.31.20.102",
"100.20.20.218"]# I've shortened the list
here for list clarity #
then {
action(type="omfile" dirCreateMode="0755" FileCreateMode="0644"
dynaFile="ESXi_app")
stop }
Is there a "priority" in how rsyslog reads/merges/loads the different configs
between the main config (/etc/rsyslog.conf) and the others like
/etc/rsyslog.d/ESXi.conf?
One other oddity kinda/sorta related. Messages like these keep writing to the
terminal:
Message from sysl...@sdcurpe02.mycompany.com at Dec 2 09:34:53 ...
localcli: libsmartsata: Not an ATA SMART
device:naa.600507680c82811eb86a
Message from sysl...@sdcurpe02.mycompany.com at Dec 2 09:34:54 ...
localcli: libsmartsata: Not an ATA SMART
device:naa.600507680c82811eb86b
Patrick Swartz
-Original Message-
From: rsyslog-boun...@lists.adiscon.com
[mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of David Lang
Sent: Thursday, December 01, 2016 3:20 PM
To: rsyslog-users
Subject: Re: [rsyslog] filters question
On Thu, 1 Dec 2016, Swartz, Patrick wrote:
> Hello,
> Confession... I'm still learning rsyslog after many years of working with
> syslog-ng. I'm using rsyslog-8.4.0-8.3 on a SLES12.1 system and am trying to
> capture my ESXi host logs.
>
> Here is my current filter for those:
>
> cat /etc/rsyslog.d/ESXi.conf
>
> template(name="ESXi_app" type="string"
> string="/var/splunk-syslog/ESXi/%FROMHOST%/%FROMHOST%-%$NOW%.log")
> if $hostname startswith ["cdcubde",
>"sdcubde",
>"sdcubpe",
>"cdcubpe",
>"cdcubdmz",
>"cdcurpe",
>"sdcurpe"]
> then {
> action(type="omfile" dirCreateMode="0755" FileCreateMode="0644"
> dynaFile="ESXi_app")
>}
> else {
> if $programname contains ["Hostd",
>"Vpxa",
>"xmlns",
>"soapenv",
>"cdcubpe02"]
> then {
>action(type="omfile" dirCreateMode="0755" FileCreateMode="0644"
> dynaFile="ESXi_app")
>}
> stop }
>
> I added the extra "else/if" because even though the 'startwith' was mostly
> working, it wasn't working 100%. And, now even with the extra else/if some
> messages are still falling through to my "Unknownl" and I don't understand
> why.
>
> Example message that is falling through -
>
> cat
> Unknown/cdcubdmz01.mycompany.com/cdcubdmz01.mycompany.com-2016-12-01.l
> og
> 2016-12-01T10:01:22.690936-05:00 cdcubdmz01.mycompay.com soapenv:
> Body> xmlns='urn:vim25'>partner ceptanceResponse>
>
> I'm using different configs in /etc/rsyslog.d/ for the different filters
> (ESXi, FireEye, PaloAlto, etc), then my Unknown filter is in the
> /etc/rsyslog.conf file. Is that approach wrong?
>
> From my rsyslog.conf:
> template(name="Unknown" type="string"
> string="/var/splunk-syslog/Unknown/%HOSTNAME%/%FROMHOST%-%$NOW%.log")
> *.* action(type="omfile" dirCreateMode="0755" FileCreateMode="0644"
> dynaFile="Unknown")
>
> Any help would be greatly appreciated.
Whenever a filter isn't working as expected, the first thing to do is to look
at what the data actually is that you are filtering against 99% of the time the
problem is that the variable doesn't contain what you expect it to.
in your 'unknown' section, log the data with the template RSYSLOG_DebugFormat
to a file and look at what it's writing.
Or, since you are just look
Re: [rsyslog] filters question
are you sure there are no other errors in your config? do rsyslogd -N1 and check
for any errors. Once you have errors in the config all best are off
David Lang
On Fri, 2 Dec 2016, Swartz, Patrick wrote:
Date: Fri, 2 Dec 2016 14:40:05 +
From: "Swartz, Patrick"
Reply-To: rsyslog-users
To: rsyslog-users
Subject: Re: [rsyslog] filters question
Okay... I've made some changes to my configs using the output from the debug. I'm now
using "fromhost_ip ==" and statically listing every possible IP in the array,
and still some messages are falling through to my Unclassified.
Probably better to show than to try and explain...
From /etc/rsyslogd.conf:
### for debug
*.* /var/splunk-syslog/msgdebug.log;RSYSLOG_DebugFormat
From debug file:
Debug line with all properties:
FROMHOST: 'sdcubpe08.mycompany.com', fromhost-ip: '100.20.20.218', HOSTNAME:
'sdcubpe08.mycompany.com', PRI: 167,
syslogtag 'Vpxa:', programname: 'Vpxa', APP-NAME: 'Vpxa', PROCID: '-', MSGID:
'-',
TIMESTAMP: 'Dec 2 14:20:09', STRUCTURED-DATA: '-',
msg: ' verbose vpxa[2ACE2B70] [Originator@6876 sub=halservices
opID=WFU-32dbe2e3] [VpxaHalServices] HostChanged Event Fired, properties
changed [runtime.healthSystemRuntime]'
escaped msg: ' verbose vpxa[2ACE2B70] [Originator@6876 sub=halservices
opID=WFU-32dbe2e3] [VpxaHalServices] HostChanged Event Fired, properties
changed [runtime.healthSystemRuntime]'
inputname: imtcp rawmsg: '<167>2016-12-02T14:20:09.131Z sdcubpe08.mycompany.com
Vpxa: verbose vpxa[2ACE2B70] [Originator@6876 sub=halservices opID=WFU-32dbe2e3]
[VpxaHalServices] HostChanged Event Fired, properties changed
[runtime.healthSystemRuntime]'
$!:
$.:
$/:
From my /etc/rsyslog.d/ESXi.conf
template(name="ESXi_app" type="string"
string="/var/splunk-syslog/ESXi/%FROMHOST%/%FROMHOST%-%$NOW%.log")
if $fromhost-ip == ["100.31.20.101",
"100.31.20.102",
"100.20.20.218"]# I've shortened the list
here for list clarity #
then {
action(type="omfile" dirCreateMode="0755" FileCreateMode="0644"
dynaFile="ESXi_app")
stop }
Is there a "priority" in how rsyslog reads/merges/loads the different configs
between the main config (/etc/rsyslog.conf) and the others like /etc/rsyslog.d/ESXi.conf?
One other oddity kinda/sorta related. Messages like these keep writing to the
terminal:
Message from sysl...@sdcurpe02.mycompany.com at Dec 2 09:34:53 ...
localcli: libsmartsata: Not an ATA SMART
device:naa.600507680c82811eb86a
Message from sysl...@sdcurpe02.mycompany.com at Dec 2 09:34:54 ...
localcli: libsmartsata: Not an ATA SMART
device:naa.600507680c82811eb86b
Patrick Swartz
-Original Message-
From: rsyslog-boun...@lists.adiscon.com
[mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of David Lang
Sent: Thursday, December 01, 2016 3:20 PM
To: rsyslog-users
Subject: Re: [rsyslog] filters question
On Thu, 1 Dec 2016, Swartz, Patrick wrote:
Hello,
Confession... I'm still learning rsyslog after many years of working with
syslog-ng. I'm using rsyslog-8.4.0-8.3 on a SLES12.1 system and am trying to
capture my ESXi host logs.
Here is my current filter for those:
cat /etc/rsyslog.d/ESXi.conf
template(name="ESXi_app" type="string"
string="/var/splunk-syslog/ESXi/%FROMHOST%/%FROMHOST%-%$NOW%.log")
if $hostname startswith ["cdcubde",
"sdcubde",
"sdcubpe",
"cdcubpe",
"cdcubdmz",
"cdcurpe",
"sdcurpe"]
then {
action(type="omfile" dirCreateMode="0755" FileCreateMode="0644"
dynaFile="ESXi_app")
}
else {
if $programname contains ["Hostd",
"Vpxa",
"xmlns",
"soapenv",
"cdcubpe02"]
then {
action(type="omfile" dirCreateMode="0755" FileCreateMode="0644"
dynaFile="ESXi_app")
}
stop }
I added the extra "else/if" because even though the 'startwith' was mostly working, it
wasn't working 100%. And, now even with the extra else/if some messages are still falling through
to my "Unknownl" and I don't understand why.
Example message that is falling through -
cat
Unknown/cdcubdmz01.mycompany.com/cdcubdmz01.mycompany.com-2016-12-01.l
og
2016-12-01T10:01:22.690936-05:00 cdcubdmz01.mycompay.com soapenv:
Body>partner
I'm using different configs in /etc/rsyslog.d/ for the different filters
(ESXi, FireEye, PaloAlto, etc), then my Unknown filter is in the
/etc/rsyslog.conf file. Is that approach wrong?
From my rsyslog.conf:
template(name="Unknown" type="string"
string="/var/splunk-syslog/Unknown/%HOSTNAME%/%FROMHOST%-%$NOW%.log")
*.* action(type="omfile" dirCreateMode="0755" FileCreateMode="0644"
dynaFile="Unknown")
Any help would be greatly appreciated.
Whenever a filter isn't working as expected, the first thing to do is to look
at what the data actually is that you are filtering against 99% of the time the
problem is that th
Re: [rsyslog] filters question
root@whqlrsyslog01 # rsyslogd -N1
rsyslogd: version 8.4.0, config validation run (level 1), master config
/etc/rsyslog.conf
rsyslogd: End of config validation run. Bye.
Here is my full rsyslogd.conf (minus comments). My additions/changes are
marked to the side (the comments are not in the file), everything else is stock
from SUSE.
root@whqlrsyslog01 # sed -e '/\s*#.*$/d' -e '/^\s*$/d' /etc/rsyslog.conf
$umask # Added by pswartz
*.* /var/splunk-syslog/msgdebug.log;RSYSLOG_DebugFormat
# Added by pswartz
$ModLoad immark.so
$MarkMessagePeriod 3600
$ModLoad imuxsock.so
$RepeatedMsgReduction on
$ModLoad imklog.so
$klogConsoleLogLevel1
$IncludeConfig /etc/rsyslog.d/*.conf # Added by pswartz
$IncludeConfig /etc/rsyslog.d/*.template # Added by
pswartz
$umask
template(name="Unclassified" type="string"
string="/var/splunk-syslog/Unclassified/%HOSTNAME%/%FROMHOST%-%$NOW%.log")
# Added by pswartz
*.* action(type="omfile" dirCreateMode="0755" FileCreateMode="0644"
dynaFile="Unclassified") # Added by pswartz
if ( \
/* kernel up to warning except of firewall */ \
($syslogfacility-text == 'kern') and \
($syslogseverity <= 4 /* warning */ ) and not \
($msg contains 'IN=' and $msg contains 'OUT=') \
) or ( \
/* up to errors except of facility authpriv */ \
($syslogseverity <= 3 /* errors */ ) and not \
($syslogfacility-text == 'authpriv') \
) \
then {
/dev/tty10
|/dev/xconsole
}
*.emerg :omusrmsg:*
if ($syslogfacility-text == 'kern') and \
($msg contains 'IN=' and $msg contains 'OUT=') \
then {
-/var/log/firewall
stop
}
if ($programname == 'acpid' or $syslogtag == '[acpid]:') and \
($syslogseverity <= 5 /* notice */) \
then {
-/var/log/acpid
stop
}
if ($programname == 'NetworkManager') or \
($programname startswith 'nm-') \
then {
-/var/log/NetworkManager
stop
}
mail.* -/var/log/mail
mail.info -/var/log/mail.info
mail.warning-/var/log/mail.warn
mail.err /var/log/mail.err
news.crit -/var/log/news/news.crit
news.err-/var/log/news/news.err
news.notice -/var/log/news/news.notice
*.=warning;*.=err -/var/log/warn
*.crit /var/log/warn
*.*;mail.none;news.none -/var/log/messages
local0.*;local1.* -/var/log/localmessages
local2.*;local3.* -/var/log/localmessages
local4.*;local5.* -/var/log/localmessages
local6.*;local7.* -/var/log/localmessages
Is there a way to also validate the configs in /etc/rsyslog.d/? Or will the -N1
also validate those?
Thank you for your help and patience,
Patrick H Swartz
-Original Message-
From: rsyslog-boun...@lists.adiscon.com
[mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of David Lang
Sent: Friday, December 02, 2016 11:55 AM
To: rsyslog-users
Subject: Re: [rsyslog] filters question
are you sure there are no other errors in your config? do rsyslogd -N1 and
check for any errors. Once you have errors in the config all best are off
David Lang
On Fri, 2 Dec 2016, Swartz, Patrick wrote:
> Date: Fri, 2 Dec 2016 14:40:05 +
> From: "Swartz, Patrick"
> Reply-To: rsyslog-users
> To: rsyslog-users
> Subject: Re: [rsyslog] filters question
>
> Okay... I've made some changes to my configs using the output from the debug.
> I'm now using "fromhost_ip ==" and statically listing every possible IP in
> the array, and still some messages are falling through to my Unclassified.
> Probably better to show than to try and explain...
>
> From /etc/rsyslogd.conf:
> ### for debug
> *.* /var/splunk-syslog/msgdebug.log;RSYSLOG_DebugFormat
>
> From debug file:
> Debug line with all properties:
> FROMHOST: 'sdcubpe08.mycompany.com', fromhost-ip: '100.20.20.218',
> HOSTNAME: 'sdcubpe08.mycompany.com', PRI: 167, syslogtag 'Vpxa:',
> programname: 'Vpxa', APP-NAME: 'Vpxa', PROCID: '-', MSGID: '-',
> TIMESTAMP: 'Dec 2 14:20:09', STRUCTURED-DATA: '-',
> msg: ' verbose vpxa[2ACE2B70] [Originator@6876 sub=halservices
> opID=WFU-32dbe2e3] [VpxaHalServices] HostChanged Event Fired, properties
> changed [runtime.healthSystemRuntime]'
> escaped msg: ' verbose vpxa[2ACE2B70] [Originator@6876 sub=halservices
> opID=WFU-32dbe2e3] [VpxaHalServices] HostChanged Event Fired, properties
> changed [runtime.healthSystemRuntime]'
> inputname: imtcp rawmsg: '<167>2016-12-02T14:20:09.131Z
> sdcubpe08.mycompany.com Vpxa: verbo
Re: [rsyslog] filters question
On Fri, 2 Dec 2016, Swartz, Patrick wrote:
root@whqlrsyslog01 # rsyslogd -N1
rsyslogd: version 8.4.0, config validation run (level 1), master config
/etc/rsyslog.conf
rsyslogd: End of config validation run. Bye.
Here is my full rsyslogd.conf (minus comments). My additions/changes are
marked to the side (the comments are not in the file), everything else is stock
from SUSE.
root@whqlrsyslog01 # sed -e '/\s*#.*$/d' -e '/^\s*$/d' /etc/rsyslog.conf
$umask # Added by pswartz
*.* /var/splunk-syslog/msgdebug.log;RSYSLOG_DebugFormat
# Added by pswartz
$ModLoad immark.so
$MarkMessagePeriod 3600
$ModLoad imuxsock.so
$RepeatedMsgReduction on
we actually recommend not doing message reduction, it's a lot easier for
monitoring to deal with the same log message 500 times than a log message that
says 'last message repeated 499 times'
$ModLoad imklog.so
$klogConsoleLogLevel1
$IncludeConfig /etc/rsyslog.d/*.conf # Added by pswartz
$IncludeConfig /etc/rsyslog.d/*.template # Added by
pswartz
$umask
template(name="Unclassified" type="string"
string="/var/splunk-syslog/Unclassified/%HOSTNAME%/%FROMHOST%-%$NOW%.log") # Added by
pswartz
*.* action(type="omfile" dirCreateMode="0755" FileCreateMode="0644"
dynaFile="Unclassified") # Added by pswartz
if ( \
are these trailing slashes in the file? they should not be needed
/* kernel up to warning except of firewall */ \
I hope this is one of the comments you say are not actually in the file.
($syslogfacility-text == 'kern') and \
($syslogseverity <= 4 /* warning */ ) and not \
($msg contains 'IN=' and $msg contains 'OUT=') \
) or ( \
/* up to errors except of facility authpriv */ \
($syslogseverity <= 3 /* errors */ ) and not \
($syslogfacility-text == 'authpriv') \
) \
then {
/dev/tty10
|/dev/xconsole
}
*.emerg :omusrmsg:*
if ($syslogfacility-text == 'kern') and \
($msg contains 'IN=' and $msg contains 'OUT=') \
then {
-/var/log/firewall
the leading - does nothing in rsyslog (in traditional syslog it meant that the
write did not need to be synchronous, but in rsyslog everything is async due to
the queues)
stop
}
if ($programname == 'acpid' or $syslogtag == '[acpid]:') and \
($syslogseverity <= 5 /* notice */) \
then {
-/var/log/acpid
stop
}
if ($programname == 'NetworkManager') or \
($programname startswith 'nm-') \
then {
-/var/log/NetworkManager
stop
}
mail.* -/var/log/mail
mail.info -/var/log/mail.info
mail.warning-/var/log/mail.warn
mail.err /var/log/mail.err
news.crit -/var/log/news/news.crit
news.err-/var/log/news/news.err
news.notice -/var/log/news/news.notice
*.=warning;*.=err -/var/log/warn
*.crit /var/log/warn
*.*;mail.none;news.none -/var/log/messages
local0.*;local1.* -/var/log/localmessages
local2.*;local3.* -/var/log/localmessages
local4.*;local5.* -/var/log/localmessages
local6.*;local7.* -/var/log/localmessages
Is there a way to also validate the configs in /etc/rsyslog.d/? Or will the -N1
also validate those?
-N1 also validates those
I am not seeing anything obvious, unless the trailing backslashes are confusing
things somehow.
but this doesn't match your e-mail blow (where is the fromhost_ip and the
array)
the messags you show below writing to the console look like they are kernel
messages, so your config is writing them out as specified.
remember, it doesn't stop processing the log at the first match, it keeps going
in case there are other matches as well.
David Lang
-Original Message-
From: rsyslog-boun...@lists.adiscon.com
[mailto:rsyslog-boun...@lists.adiscon.com] On Behalf Of David Lang
Sent: Friday, December 02, 2016 11:55 AM
To: rsyslog-users
Subject: Re: [rsyslog] filters question
are you sure there are no other errors in your config? do rsyslogd -N1 and
check for any errors. Once you have errors in the config all best are off
David Lang
On Fri, 2 Dec 2016, Swartz, Patrick wrote:
Date: Fri, 2 Dec 2016 14:40:05 +
From: "Swartz, Patrick"
Reply-To: rsyslog-users
To: rsyslog-users
Subject: Re: [rsyslog] filters question
Okay... I've made some changes to my configs using the output from the debug. I'm now
using "fromhost_ip ==" and statically listing every possible IP in the array,
and still some messages are falling through to
