Re: [rsyslog] liblognorm vs grok

2016-12-07 Thread mosto...@gmail.com
a literal space is always more efficent than whitespace, only use whitespace if there can be more than one space, or tabs Ok. just a note, the new syntax is not always better than the old syntax 127.0.0.1 - - [17/Mar/2016:18:15:06 +0100] "GET /redacted HTTP/1.1" 200 59506 type=@apache_com

Re: [rsyslog] liblognorm vs grok

2016-12-07 Thread mosto...@gmail.com
that is the same type of bug, just for another type. just add a note that we need to allow end of line for all types, it's not limited to space. I'm missing code commenting...probably I'm going to switch back to ~doc tasks :P ___ rsyslog mailing l

Re: [rsyslog] liblognorm vs grok

2016-12-07 Thread mosto...@gmail.com
{"type":"@apache" name="."} ? actuall, %{"type":"@apache" name="."}% This is one of the places where I like to use the older, more compact syntax :-) Older/Compact doesn't seem to have an alternative, reason why I started using JSON syntax...right?

Re: [rsyslog] liblognorm vs grok

2016-12-07 Thread mosto...@gmail.com
I'm still trying to reproduce/understand what is happening and building a test case for the github issue if needed. Consider the following HTTP access lines: 127.0.0.1 - - [17/Mar/2016:18:15:06 +0100] "GET /redacted HTTP/1.1" 200 59506 127.0.0.1 - - [17/Mar/2016:18:15:24 +0100] "OPTION

Re: [rsyslog] liblognorm vs grok

2016-12-07 Thread mosto...@gmail.com
I think it's a problem, several of the types require a space at the end, and I think they should all be modified to allow either a space or a end-of-line. ack. It's on my list for early next year. better check if one exists, I also think David created one. This is for the liblognorm project.

Re: [rsyslog] liblognorm vs grok

2016-12-07 Thread mosto...@gmail.com
when troubleshooting things like this, create a rule file that is as minimal as you can get and parse with the -v option, it will show you what it's doing as it walks through the line. I don't see how it parsed each message. Perhaps a debug option must be enabled? number of tree nodes:

Re: [rsyslog] liblognorm vs grok

2016-12-07 Thread mosto...@gmail.com
almost, %@apache% makes no more sense than %word%, you need to give the match a name so %log:@apache% would work, or if you want to move everything up a later (rather than having $!apache!ip) you could do %.:@apache% That should work How would that be using JSON syntax? {"type":"@apache"

Re: [rsyslog] liblognorm vs grok

2016-12-07 Thread mosto...@gmail.com
Should something like this work? {"type":"@apache"}, {"type":"alternative","parser":[ {}, { {"type":"whitespace"}, ... } ]} El 07/12/16 a las 11:08, Rainer Gerhards

Re: [rsyslog] liblognorm vs grok

2016-12-07 Thread mosto...@gmail.com
In this case, I seem to remember that number is defined as being followed by a space, so you can't use it if the number is followed by a newline. I'll have to confirm that...but may I know why? Should I fill an issue if it's indeed that way? I think it's a problem, several of the types requir

Re: [rsyslog] liblognorm vs grok

2016-12-07 Thread mosto...@gmail.com
when troubleshooting things like this, create a rule file that is as minimal as you can get and parse with the -v option, it will show you what it's doing as it walks through the line. Ok :) In this case, I seem to remember that number is defined as being followed by a space, so you can't

Re: [rsyslog] liblognorm vs grok

2016-12-05 Thread mosto...@gmail.com
I forgot: With provided rule file...why I'm getting a bunch of this errors when using /usr/lib/lognorm/lognormalizer? { "originalmsg": "127.0.0.1 - - [17\/Mar\/2016:18:15:31 +0100] \"GET \/redacted\/page HTTP\/1.1\" 200 1234", "unparsed-data&

Re: [rsyslog] liblognorm vs grok

2016-12-05 Thread mosto...@gmail.com
Hi Coming back to liblognorm, I have a few questions I'll love an expert reply. 0:D *- Documentation [1] states how to define a type, but not how to use it. Are we properly using defined type "apache" in the configuration below?* - Apache access log seem to have 2 formats: common and comb

Re: [rsyslog] liblognorm vs grok

2016-12-05 Thread mosto...@gmail.com
Is that documentation stored on a github like rsyslog's? http://www.liblognorm.com/files/manual/index.html El 05/12/16 a las 11:15, David Lang escribió: On Mon, 5 Dec 2016, mosto...@gmail.com wrote: Hi. Is there an online liblognorm tester to check the rules we are writing? Othe

Re: [rsyslog] liblognorm vs grok

2016-12-05 Thread mosto...@gmail.com
Hi. Is there an online liblognorm tester to check the rules we are writing? Otherwise, could you provide a testing guide (http://www.liblognorm.com/files/manual/installation.html#testing) to build lognormalizer to test? El 04/10/16 a las 19:27, mosto...@gmail.com escribió: Hi Radu After

Re: [rsyslog] REK stack

2016-12-02 Thread mosto...@gmail.com
Rek Project makes us sound like dangerous anarchists, or possibly a dub-techno outfit. LOL ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https:/

Re: [rsyslog] REK stack

2016-12-02 Thread mosto...@gmail.com
TREK (long journey) seems a more friendly name for me. It also stands for "(The) Rsyslog ElasticSearch Kibana" project/stack :P (I can't stop posting on this list) El 02/12/16 a las 11:48, Rainer Gerhards escribió: Hi all, I start a new thread as the other one has a million of different topi

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

2016-12-02 Thread mosto...@gmail.com
ainer Gerhards escribió: 2016-12-02 11:45 GMT+01:00 mosto...@gmail.com : El 02/12/16 a las 10:50, David Lang escribió: remember that values in logs may be under the control of an attacker (just about every field can be hand-crafted by a person sending you a log message). I know you are thinking o

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

2016-12-02 Thread mosto...@gmail.com
El 02/12/16 a las 10:50, David Lang escribió: remember that values in logs may be under the control of an attacker (just about every field can be hand-crafted by a person sending you a log message). I know you are thinking of the simple case where you are only reading files and processing them,

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

2016-12-02 Thread mosto...@gmail.com
(as an optimization, make the most common apps the earliest in the directory) And that's where that /if exists call $!tag/ idea makes more sense, cause you don't need to compare, neither to arrange most used applications at top. ___ rsyslog mailing

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

2016-12-02 Thread mosto...@gmail.com
Hi Combined threads reply follow. On Thu, 1 Dec 2016, David Lang wrote sigh, this is getting a wee bit frustrating, you keep saying "it hurts when I do X", we say "that doesn't work well, do Y" and you come back a day or so later saying "but it really huts when I do X"... (it doesn't help w

Re: [rsyslog] Are we building an ERK stack?

2016-12-02 Thread mosto...@gmail.com
El 01/12/16 a las 23:08, David Lang escribió: On Thu, 1 Dec 2016, mosto...@gmail.com wrote: I think that you are going to end up with some grief, if the message could not be insterted into ES for some reason, I think the odds are good that you will find that rawmsg can't be inserted e

Re: [rsyslog] rsyslog-doc github include rst?

2016-12-01 Thread mosto...@gmail.com
n its availability. Rainer Sent from phone, thus brief. Am 01.12.2016 19:52 schrieb "mosto...@gmail.com" : within github I guessed sphinx would do it for web-fronted, but I was asking for github. El 01/12/16 a las 19:50, Rainer Gerhards escribió: The headers etc are actually gene

Re: [rsyslog] rsyslog-doc github include rst?

2016-12-01 Thread mosto...@gmail.com
f a generation run. Let me know if this helps or more detail is required. Rainer Sent from phone, thus brief. Am 01.12.2016 19:12 schrieb "mosto...@gmail.com" : Hi @radu-gheorghe @rgerhards: Is there any way to include a rst document into another within github? I have been trying /r

Re: [rsyslog] Are we building an ERK stack?

2016-12-01 Thread mosto...@gmail.com
.com/blog/rek-it/ I wrote this up earlier. On Wed, 23 Nov 2016 at 19:38 mosto...@gmail.com wrote: Working, spamming mail list and writing on wiki at the same time. A lovely afternoon... Please, add your lines: https://github.com/rsyslog/rsyslog/wiki ___

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

2016-12-01 Thread mosto...@gmail.com
yup, thx. IMHO makes sense. Let's wait what David says, but I think it would make sense to open an issue refering to this thread. ... I know, I also must find time to actually work on some of them... ;-) This is our current /core.conf/ draft: global( MaxMessageSize="32k" wor

[rsyslog] rsyslog-doc github include rst?

2016-12-01 Thread mosto...@gmail.com
Hi @radu-gheorghe @rgerhards: Is there any way to include a rst document into another within github? I have been trying /raw/ and other directives without success (seems due to security concerns) (Trying to include legal foot/license for every page) Thanks. __

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

2016-12-01 Thread mosto...@gmail.com
El 01/12/16 a las 19:04, Rainer Gerhards escribió: 2016-12-01 18:56 GMT+01:00 mosto...@gmail.com : El 01/12/16 a las 18:37, Rainer Gerhards escribió: 2016-12-01 18:33 GMT+01:00 mosto...@gmail.com : Hi Is there any way to dynamically invoke a ruleset? eg: call $var (I'm trying to

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

2016-12-01 Thread mosto...@gmail.com
El 01/12/16 a las 18:37, Rainer Gerhards escribió: 2016-12-01 18:33 GMT+01:00 mosto...@gmail.com : Hi Is there any way to dynamically invoke a ruleset? eg: call $var (I'm trying to avoid having +200 if statements... not yet, but 90% sure evrything is in place to make implementation easy.

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

2016-12-01 Thread mosto...@gmail.com
Hi Is there any way to dynamically invoke a ruleset? eg: call $var (I'm trying to avoid having +200 if statements... El 25/11/16 a las 14:13, David Lang escribió: On Fri, 25 Nov 2016, mosto...@gmail.com wrote: I may be confused about which part is on the sender and which part is o

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

2016-12-01 Thread mosto...@gmail.com
rector, Technical Product Management On Dec 1, 2016, at 8:28 AM, mosto...@gmail.com<mailto:mosto...@gmail.com> wrote: After meal, as usually happens, those quotes sparkled. Doesn't rsyslog conf grammar allows that neither? /(eg: set $!data!foo="this_doesnt_seem_to_work_$programn

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

2016-12-01 Thread mosto...@gmail.com
After meal, as usually happens, those quotes sparkled. Doesn't rsyslog conf grammar allows that neither? /(eg: set $!data!foo="this_doesnt_seem_to_work_$programname";)/ El 01/12/16 a las 14:57, mosto...@gmail.com escribió: This worked, but I have lost 2 hours and still do

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

2016-12-01 Thread mosto...@gmail.com
This worked, but I have lost 2 hours and still don't see where's the evil. Works: module(load="omrelp") ruleset(name="relp") { set $!data!group=field($programname,47,1); set $!data!msg=$msg; action( action.reportSuspension="on" action.resumeR

Re: [rsyslog] about imfile

2016-12-01 Thread mosto...@gmail.com
A message without TAG (malformed RFC 3164 message), no matter if it's read from file or it arrives from socket, won't have a tag Hence, setting it only for imfile won't fix it for socket modules. I am not ready for this discussion again. In rsyslog, rfc3164 messages always have a tag. See previ

Re: [rsyslog] about imfile

2016-12-01 Thread mosto...@gmail.com
now, that makes sense! :D Thanks El 01/12/16 a las 13:06, Rainer Gerhards escribió: 2016-12-01 12:55 GMT+01:00 Rainer Gerhards : 2016-12-01 11:54 GMT+01:00 mosto...@gmail.com : because a syslog message contains tag. mind-blowing explanation :P Well, as the property is already there, why

Re: [rsyslog] about imfile

2016-12-01 Thread mosto...@gmail.com
El 01/12/16 a las 12:55, Rainer Gerhards escribió: 2016-12-01 11:54 GMT+01:00 mosto...@gmail.com : because a syslog message contains tag. mind-blowing explanation :P Well, as the property is already there, why would you like to have a config parameter for something that by definition will

Re: [rsyslog] about imfile

2016-12-01 Thread mosto...@gmail.com
read modes other than 0 currently seem to have issues in inotify mode Any open issues? it's an based-on-experienced-warning message? legacy? I am not aware of one, which does not necessarily mean none exists. So you need to check the issue trackers :-( The longer-term question is if we s

Re: [rsyslog] about imfile

2016-12-01 Thread mosto...@gmail.com
El 30/11/16 a las 22:51, David Lang escribió: On Wed, 30 Nov 2016, mosto...@gmail.com wrote: According to documentation: State files are used to track which parts of the monitored file are already processed. Do state files keep just "last reading position" or as doc suggests

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

2016-12-01 Thread mosto...@gmail.com
47 3421.997714715:main Q:Reg/w0 : 2 3421.997728731:main Q:Reg/w0 : END SET El 30/11/16 a las 22:32, Rainer Gerhards escribió: Can you show what the properties are for this message? Sent from phone, thus brief. Am 30.11.2016 20:18 schrieb "mosto...@gmail.com" : Hi

[rsyslog] about imfile

2016-11-30 Thread mosto...@gmail.com
According to documentation: State files are used to track which parts of the monitored file are already processed. Do state files keep just "last reading position" or as doc suggests a file can be processed in multiple chunks(parts)? Note that when $WorkDirectory is not set or set to a

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

2016-11-30 Thread mosto...@gmail.com
Hi I'm still not able to get it, and *perhaps reproduced an issue related to properties not being accessible*... Consider the following relay.conf: global( MaxMessageSize="32k" workDirectory="/var/spool/rsyslog" parser.escapeControlCharactersOnReceive="off" ) template(

Re: [rsyslog] auto-generating module documentation

2016-11-30 Thread mosto...@gmail.com
work (at least a couple of days), and boils down to that everyone but developers loses the ability to write module doc. Rainer Sent from phone, thus brief. Am 30.11.2016 09:36 schrieb "David Lang" : On Wed, 30 Nov 2016, mosto...@gmail.com wrote: Probably a stupid idea...will

Re: [rsyslog] auto-generating module documentation

2016-11-30 Thread mosto...@gmail.com
Probably a stupid idea...will it make sense that information being populated from rsysloc-doc? (or viceversa, like javadoc) Having everything on one place looks great to me El 28/11/16 a las 17:50, David Lang escribió: we have a few cases where the documentation doesn't match the module param

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

2016-11-25 Thread mosto...@gmail.com
I may be confused about which part is on the sender and which part is on the receiver. sender: a bunch of imfiles forwarded using RELP receiver: receives a JSON with msg=plain/original message, normalize and extract fields (that should be added to JSON) each "application" should define his ow

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

2016-11-25 Thread mosto...@gmail.com
string="<%pri%>%timestamp:::date-rfc3339% %hostname% logs/$!data!group/$!data!app: %$!data%") to be fully correct Done. also watch out, the programname is limited to 32 characters, don't let your group and app names get too long. Wasn't it possible to change that? IIRC we had some issues with

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

2016-11-25 Thread mosto...@gmail.com
What about...? *remote.conf*(Please, notice there are commented questions) global( MaxMessageSize="32k" ) template(name="json" type="string" string="%timestamp:::date-rfc3339% %hostname% logs/$!data!group/$!data!app %$!data%") module(load="omrelp") ruleset(name="re

Re: [rsyslog] rsyslog fails to start due to high queue

2016-11-25 Thread mosto...@gmail.com
ebuild with the information you have given but I am still not sure of how they got corrupted and what happened to the ones that were in memory, was everything saved on the hard drive or was there information that was lost. Kind Regards, Kosta On Fri, Nov 25, 2016 at 3:42 AM, David Lang wrote: On Fri, 25

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

2016-11-25 Thread mosto...@gmail.com
After sleeping on it, I'm still thinking about "defining a separate pipeline for each application". To sum up, each application could do his own thing and return the message to the queue, in order to be processed by other modules...until it's done, and indexed into ES Is there a way to put the a

Re: [rsyslog] Are we building an ERK stack?

2016-11-25 Thread mosto...@gmail.com
Thanks! It's your mmdblookup opensourced? El 25/11/16 a las 03:46, chenlin rao escribió: re-upload an english version. The content was a little old though. 2016-11-23 22:39 GMT+08:00 mosto...@gmail.com : http://www.slideshare.net/chenryn/elk-stack-at-weibocom I NEED the english versi

Re: [rsyslog] rsyslog fails to start due to high queue

2016-11-25 Thread mosto...@gmail.com
TBH, it depends if you prefer building from source or instability Kidding away, I know there are problems in older versions, they are Isn't adiscon repo valid? ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsysl

Re: [rsyslog] making config changes to a running rsyslog

2016-11-24 Thread mosto...@gmail.com
what are people's thoughts on these ideas? Notice there can be multiple reload scenarios: - reload rsyslog config (new modules, inputs, rulesets, actions...) - add new inputs - modify a template - resize a queue The simplest approach I can imagine is to signal HUP to reload: when signal

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

2016-11-24 Thread mosto...@gmail.com
Once I use it ;) Yet have to change overview to RST... El 24/11/16 a las 17:36, Rainer Gerhards escribió: 2016-11-24 17:21 GMT+01:00 mosto...@gmail.com : # I would consider adding a section here to look for parsing failures and log them to someplace for later investigation, probably in raw

Re: [rsyslog] Are we building an ERK stack?

2016-11-24 Thread mosto...@gmail.com
El 24/11/16 a las 17:42, Rainer Gerhards escribió: I added a project to rsyslog on github, where we can bind Issues to: shouldn't that work for documentation? (as commented previously, I would love to have 1 repo!) https://github.com/rsyslog/rsyslog/projects/1 I guess I must make the relatio

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

2016-11-24 Thread mosto...@gmail.com
this way you use the tag that you set on the input line to create the metadata fields you want with only one ruleset for all the app inputs. cool! The other thing you could do is to create a table lookup to map the programname (the value you set as the tag in the input) to the values you wan

Re: [rsyslog] Are we building an ERK stack?

2016-11-24 Thread mosto...@gmail.com
or docker swarm mode :D El 24/11/16 a las 16:22, David Lang escribió: On Thu, 24 Nov 2016, mosto...@gmail.com wrote: As we are concerned about high availability and load balancing, we plan to deploy multiple instances. just a note that while rsyslog doesn't implement load balancing i

Re: [rsyslog] Are we building an ERK stack?

2016-11-24 Thread mosto...@gmail.com
not really, but we haven't had anyone experiment with thousands of them, so it's possible, but unlikely that there would be a measureable slowdown as rsyslog finds the right one to use. The bigger overhead is in interpreting the template, that's where simplifying it to be $! or $!foo would b

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

2016-11-24 Thread mosto...@gmail.com
Last hours have been quite busy! After reading your comments, here's where we are. Please, advice and comment. On some remote hosts, logs are readed from files. Each remote will have a configuration like: # remote.conf global( MaxMessageSize="32k" ) template(name="j

Re: [rsyslog] Are we building an ERK stack?

2016-11-24 Thread mosto...@gmail.com
rlier. On Wed, 23 Nov 2016 at 19:38 mosto...@gmail.com wrote: Working, spamming mail list and writing on wiki at the same time. A lovely afternoon... Please, add your lines: https://github.com/rsyslog/rsyslog/wiki ___ rsyslog mailing list http://lists.a

Re: [rsyslog] Coding doc

2016-11-23 Thread mosto...@gmail.com
I mean https://github.com/rsyslog/rsyslog-doc/pull/188 El 23/11/16 a las 21:25, Rainer Gerhards escribió: Sent from phone, thus brief. Am 23.11.2016 21:01 schrieb "mosto...@gmail.com" : What did happen to the "new documentation format" effort? It's well alive an

Re: [rsyslog] Coding doc

2016-11-23 Thread mosto...@gmail.com
What did happen to the "new documentation format" effort? I think wiki could be fair enough, but I don't know how it will behave for old-releases. If I'm not wrong, current system allow to have different documentation pages according to each version, so perhaps it worths keep it. On the other

Re: [rsyslog] Are we building an ERK stack?

2016-11-23 Thread mosto...@gmail.com
Working, spamming mail list and writing on wiki at the same time. A lovely afternoon... Please, add your lines: https://github.com/rsyslog/rsyslog/wiki ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/pr

Re: [rsyslog] Feedback request: minimal log shipper project

2016-11-23 Thread mosto...@gmail.com
do you have an example of what was sent over the wire and how it was mishandled? It may be that we need to tweak the default parsers. https://github.com/rsyslog/rsyslog/issues/1191 no need to convert, just add the ability to use the slower, but more flexible mmnormalize. You said it has bug

Re: [rsyslog] Are we building an ERK stack?

2016-11-23 Thread mosto...@gmail.com
The problem is the fact that there are so many ways timestamp data can be scattered in a log message. take a look at the output of date --help and look at all the formatting options. I guarantee that some log somewhere will use every one of them. IIRC, you had found a solution to this... ___

Re: [rsyslog] A couple of questions regarding rsyslog debug.

2016-11-23 Thread mosto...@gmail.com
yup, the function is named DBGPRINTF() and some helpers. Being used since ~1970 (the original dbgprintf() call stems back to the original code ;)). Sometimes we use #ifdef DEBUG, but only if we actually need to compute something beyond simple output generation (e.g. iterate over a structure). Th

Re: [rsyslog] Are we building an ERK stack?

2016-11-23 Thread mosto...@gmail.com
you can combine variables to form a string that looks like a date in the output, but you can't take arbitrary date parts in a log message and parse them into a real timestamp field that would let you output it in different formats. back on my pipeline proposal, wouldn't this solve the issue?

Re: [rsyslog] omriemann Re: Are we building an ERK stack?

2016-11-23 Thread mosto...@gmail.com
As main promoter (ring the bell and run like hell), could you make some tests comparing filebeat vs imfile performance and footprint? El 23/11/16 a las 19:32, Bob Gregory escribió: I can easily enough knock together an omriemann - it's protobuf over TCP or UDP. TCP allows for message ack. Th

Re: [rsyslog] Feedback request: minimal log shipper project

2016-11-23 Thread mosto...@gmail.com
We both reached an agreement about exceptions and I'm happy with that :) Actually my main concern is, each time I read a line like: CHKiRet(statsobj.Construct(&pThis->statsobj)); I have to start digging what CHKiRet is, or what the hell is: CODE_STD_STRING_REQUESTparseSelectorAct(1) Any

Re: [rsyslog] Feedback request: minimal log shipper project

2016-11-23 Thread mosto...@gmail.com
why add the overhead of transporting the data between machines if you don't need to? Rsyslog is FAST, it routinely does things on one machine that other systems need many farms of machines to do. adding more machines and network connectivity will add complexity and delays, reducing performan

Re: [rsyslog] Are we building an ERK stack?

2016-11-23 Thread mosto...@gmail.com
The ugly way to do this would be a series of if $programname = "group/appX" then set $.owner = "\"group\":\"group\","\"unit\":\"unit\",\"app\":\"appX\","; do having multiple templates affect performance? (what I really noticed it's they affect loading time!) a far more elegant way to do th

Re: [rsyslog] mmnormalize with mutiple input: conditionals?

2016-11-23 Thread mosto...@gmail.com
My recommended architecture [1] is to have a local relay picking up logs from each network/datacenter, add useful metadata (fromhost-ip, what environment this is, which relay processed it, timestamp of when the log was processed on the relay, etc) and then forward the message to a central log

Re: [rsyslog] Are we building an ERK stack?

2016-11-23 Thread mosto...@gmail.com
Hi all In order to improve the first draft of ERK project, I would like to get some feedback from you. What features are you missing/you think it may be improved in rsyslog? Please, try to be as more clear/self-explanatory/simple as you can for better understanding. * logstash memory foot

Re: [rsyslog] Feedback request: minimal log shipper project

2016-11-23 Thread mosto...@gmail.com
Does this help? Although I had all these clear, reading it helped me to better understand. Thank you. A few questions appear: * You talk about threads...may we considered a distributed application architecture/other process reading that queue? That could be done with a redis/kafka rel

Re: [rsyslog] Feedback request: minimal log shipper project

2016-11-23 Thread mosto...@gmail.com
1:00 mosto...@gmail.com : https://en.wikipedia.org/wiki/Erk ...I know you can do it better... :P El 23/11/16 a las 17:34, Rainer Gerhards escribió: ERK 2016-11-23 17:34 GMT+01:00 mosto...@gmail.com : Let me start writing, cause it may be helpful as documentation too! Please do. I still need

Re: [rsyslog] Feedback request: minimal log shipper project

2016-11-23 Thread mosto...@gmail.com
https://en.wikipedia.org/wiki/Erk ...I know you can do it better... :P El 23/11/16 a las 17:34, Rainer Gerhards escribió: ERK 2016-11-23 17:34 GMT+01:00 mosto...@gmail.com : Let me start writing, cause it may be helpful as documentation too! Please do. I still need a codename! Sorry for

Re: [rsyslog] Feedback request: minimal log shipper project

2016-11-23 Thread mosto...@gmail.com
Let me start writing, cause it may be helpful as documentation too! Please do. I still need a codename! Sorry for that...but I NEED IT! ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional

Re: [rsyslog] Feedback request: minimal log shipper project

2016-11-23 Thread mosto...@gmail.com
That's a permission issue: We need to be much more restrictive (security) with who has permissions to the code than to the doc. Thus we have two repos. I'd prefer a single one, too, but that's not possible. Understood...does this happened in real life or just in paper? :P I mean: if there are r

Re: [rsyslog] Feedback request: minimal log shipper project

2016-11-23 Thread mosto...@gmail.com
Totally agreed with your comments. That's why I want a collaborative doc, where I can randomly write and you'll wisely answer. Let me start writing, cause it may be helpful as documentation too! ___ rsyslog mailing list http://lists.adiscon.net/mailma

Re: [rsyslog] Feedback request: minimal log shipper project

2016-11-23 Thread mosto...@gmail.com
That's a permission issue: We need to be much more restrictive (security) with who has permissions to the code than to the doc. Thus we have two repos. I'd prefer a single one, too, but that's not possible. Understood...does this happened in real life or just in paper? :P I mean: if there are r

Re: [rsyslog] Feedback request: minimal log shipper project

2016-11-23 Thread mosto...@gmail.com
I don't think a fork make sense, why keep two versions? As a branch in rsyslog-repo, that makes a lot of sense, but I'd move in smaller refactoring steps and merge each one as early as possible. A total rewrite from scratch is out of question, except if somone has a couple of month to years time

Re: [rsyslog] Feedback request: minimal log shipper project

2016-11-23 Thread mosto...@gmail.com
Open question to the community: does it really make sense to create a new repo at this stage? I'll try to start as clean as possible, but I agree I can start writing anywhere Name + Scope I mean this project will NOT host rsyslog code, so what will it host? Shouldn't we benefit from rsyslo

Re: [rsyslog] Feedback request: minimal log shipper project

2016-11-23 Thread mosto...@gmail.com
thus brief. Am 23.11.2016 16:03 schrieb "David Lang" : On Wed, 23 Nov 2016, mosto...@gmail.com wrote: Could any of you, aware of rsyslog internals, provide a design diagram/architecture view of how the components are working? Perhaps you could use something like: https://docs.go

Re: [rsyslog] Are we building an ERK stack?

2016-11-23 Thread mosto...@gmail.com
. If not, what would give you the Mets data? Rainer Sent from phone, thus brief. Am 23.11.2016 15:51 schrieb "mosto...@gmail.com" : there are probably ways to simplify the configs, 5K lines of configs seems excessive :-) how much of this is rulebase config vs rsyslog config?

Re: [rsyslog] Are we building an ERK stack?

2016-11-23 Thread mosto...@gmail.com
there are probably ways to simplify the configs, 5K lines of configs seems excessive :-) how much of this is rulebase config vs rsyslog config? Each app generates app-access.log, app-tomcat.log, app-application.log files. imfile allow me to add filename as metadata, but nothing more. As each

Re: [rsyslog] Are we building an ERK stack?

2016-11-23 Thread mosto...@gmail.com
http://www.slideshare.net/chenryn/elk-stack-at-weibocom I NEED the english version :P ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitte

Re: [rsyslog] Are we building an ERK stack?

2016-11-23 Thread mosto...@gmail.com
+1 Our current scenario (dockerized!): imfile_forwarder-->imrelp-->rsyslog-->redis-->logstash(grok+geoip)-->elastic We are using redis as memory buffer and to split into multiple channels/lists (using dynakey ATM). We see kafka on the horizon. We are also using several logstash container

Re: [rsyslog] Feedback request: minimal log shipper project

2016-11-23 Thread mosto...@gmail.com
./configure --help shows these options. I'm wondering if having a minimal build that created a different binary, used a different config and stripped everything down would be useful Didn't notice I could use --disable-whatever on those features having [default=yes]. Anyway, imjournal is disa

[rsyslog] mmnormalize with mutiple input: conditionals?

2016-11-23 Thread mosto...@gmail.com
Hi Considering our imrelp module receives a lot of messages from different applications/hosts (with different TAGs), what would be the best way to split/forward each message to proper ruleset/parser? Should I use /_if/_? Does http://www.rsyslog.com/doc/v8-stable/configuration/modules/mmnor

[rsyslog] A couple of questions regarding rsyslog debug.

2016-11-23 Thread mosto...@gmail.com
Hi In the past I have to build a few applications with C/C++ and I found #ifdef directives everywhere to be ugly and confusing. For example, using ifdef DEBUG to enable/disable debug messages. That's why I opt to use functions which behavior was defined depending on compile headers/source f

Re: [rsyslog] Feedback request: minimal log shipper project

2016-11-23 Thread mosto...@gmail.com
Could any of you, aware of rsyslog internals, provide a design diagram/architecture view of how the components are working? Perhaps you could use something like: https://docs.google.com/presentation/d/1UJSO0t2J6mrIyBNSkCUvrR5Q4RWbLcOof2WgB2_qA3g/edit#slide=id.g13bdc00843_0_3 I think, apart fr

Re: [rsyslog] Would imhiredis make sense?

2016-11-23 Thread mosto...@gmail.com
Logstash needs something like redis because it can't do any queueing itself. Rsyslog is built around queues, and has the ability to create multiple queues and piplines internally, you don't need to run multiple instances. I want multiples instances in order to: * Being able to process pipeli

Re: [rsyslog] Would imhiredis make sense?

2016-11-22 Thread mosto...@gmail.com
What sort of log volume are you talking about here? (logs/sec type of thing) From 0 to thousand-thousands/sec Logstash needs something like redis because it can't do any queueing itself. Rsyslog is built around queues, and has the ability to create multiple queues and piplines internally, yo

Re: [rsyslog] Feedback request: minimal log shipper project

2016-11-22 Thread mosto...@gmail.com
Quite interested on this thread http://blog.gerhards.net/2016/11/would-creating-simple-linux-log-file.html IMHO: the "issue" we're having with rsyslog is understanding the code rather than performance. Why you'll invest time doing a "faster reader" if you already have a "fast-enough processo

[rsyslog] Would imhiredis make sense?

2016-11-22 Thread mosto...@gmail.com
Hi We've been playing with logstash, rsyslog and redis for a while in order to *index into elasticsearch a bunch of application logs*. Briefly: app1-file1.log, app1-file2.log...appN-fileX.log -> pipeline -> elasticsearch. So far, we are using *redis queues and _each application_ processing

Re: [rsyslog] lowercasing structured-data SD names

2016-11-16 Thread mosto...@gmail.com
re created lowercase? Regards El 16/11/16 a las 16:05, Rainer Gerhards escribió: 2016-11-16 15:38 GMT+01:00 mosto...@gmail.com : A quick look up returned: https://github.com/rsyslog/rsyslog/search?utf8=%E2%9C%93&q=tolower This is relevant: https://github.com/rsyslog/rsyslog/issues/481 Rainer

Re: [rsyslog] lowercasing structured-data SD names

2016-11-16 Thread mosto...@gmail.com
A quick look up returned: https://github.com/rsyslog/rsyslog/search?utf8=%E2%9C%93&q=tolower El 16/11/16 a las 11:48, mosto...@gmail.com escribió: Rainer, what do you think? A "removed-lowercase" branch could work for this, while helping us to solve #1116 El 16/11/16 a la

Re: [rsyslog] lowercasing structured-data SD names

2016-11-16 Thread mosto...@gmail.com
Rainer, what do you think? A "removed-lowercase" branch could work for this, while helping us to solve #1116 El 16/11/16 a las 11:11, David Lang escribió: On Wed, 16 Nov 2016, mosto...@gmail.com wrote: According to https://tools.ietf.org/html/rfc5424#section-6.3.2

Re: [rsyslog] structured data parsing

2016-11-16 Thread mosto...@gmail.com
his directly so that if jsonroot is set to '', the fields become top-level objects. David Lang On Wed, 16 Nov 2016, mosto...@gmail.com wrote: Date: Wed, 16 Nov 2016 09:45:08 +0100 From: "mosto...@gmail.com" Reply-To: rsyslog-users To: rsyslog@lists.adiscon.com Subject: Re:

Re: [rsyslog] structured data parsing

2016-11-16 Thread mosto...@gmail.com
To be more clear: seems by default jsonroot = NULL, which IMHO should use "root as parent", instead of "rfc5424-sd". Having "rfc5424-sd" as default it's mandatory for backwards compatibility? El 16/11/16 a las 09:31, mosto...@gmail.com escribió: Seems

Re: [rsyslog] structured data parsing

2016-11-16 Thread mosto...@gmail.com
cribió: On Tue, 15 Nov 2016, mosto...@gmail.com wrote: According to http://www.rsyslog.com/doc/v8-stable/configuration/modules/mmpstrucdata.html, structure data ID and parameters are stored under *rfc5424-sd* element in JSON. It's there any way to store them as /firstchild/ elements (not

[rsyslog] lowercasing structured-data SD names

2016-11-16 Thread mosto...@gmail.com
Hi According to https://tools.ietf.org/html/rfc5424#section-6.3.2 and https://tools.ietf.org/html/rfc5424#section-6.3.3 both are *case-sensitive*. As stated in http://www.rsyslog.com/doc/v8-stable/configuration/modules/mmpstrucdata.html : "Note that the case of the RFC5424 parameter names h

[rsyslog] structured data parsing

2016-11-15 Thread mosto...@gmail.com
Hi According to http://www.rsyslog.com/doc/v8-stable/configuration/modules/mmpstrucdata.html, structure data ID and parameters are stored under *rfc5424-sd* element in JSON. It's there any way to store them as /firstchild/ elements (not using rfc5424-sd as parent, but !) ? Otherwise, wha

  1   2   >