Re: [rsyslog] Send logs in JSON format to Elasticsearch

2017-02-23 Thread mostolog--- via rsyslog
I dont know if this is what you are talking http://www.rsyslog.com/doc/master/configuration/modules/mmjsonparse.html BTW: whats your rsyslog omelasticsearch index rate? Last test I made (with a basic configuration) we got ~10k/min only with latest ES version (perhaps its not fully compatible).

[rsyslog] Using rsyslog to read docker log files

2017-02-23 Thread Chaloulos, Klearchos (Nokia - GR/Athens) via rsyslog
Hello, I am investigating whether I can use rsyslog to read from docker log files. Docker log files display logs in the following format: {"stream":"stdout","time":"2017-02-23T10:58:26.20407842Z","log":"The log message"} So I used the mmjsonparse to parse the above message and get the "log" pa

Re: [rsyslog] Send logs in JSON format to Elasticsearch

2017-02-23 Thread David Lang
lognorm has the json type that you can use to extract json from the message, or if the message is all json you can use mmjsonparse. if you aren't already using the version=2 features, you should really take a look at them, there is a lot of new capabilities in recent versions. David Lang El

Re: [rsyslog] Using rsyslog to read docker log files

2017-02-23 Thread David Lang
you can use mmnormalize and create a rulebase that uses the json type against rawmsg. David Lang On Thu, 23 Feb 2017, Chaloulos, Klearchos (Nokia - GR/Athens) via rsyslog wrote: Date: Thu, 23 Feb 2017 11:48:01 + From: "Chaloulos, Klearchos (Nokia - GR/Athens) via rsyslog" To: rs

Re: [rsyslog] rsyslog 8.25.0 (v8-stable) released

2017-02-23 Thread Thomas Deutschmann via rsyslog
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Hi, the DOC tarball [1] is missing. [1] http://www.rsyslog.com/files/download/rsyslog/rsyslog-doc-8.25.0.tar.gz - -- Regards, Thomas -BEGIN PGP SIGNATURE- Version: GnuPG v2.0 iQJ8BAEBCgBmBQJYrxOtXxSAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25z

Re: [rsyslog] Send logs in JSON format to Elasticsearch

2017-02-23 Thread Alec Swan via rsyslog
For some reason I stopped receiving emails from the mailing list even though I can see them in the archive. In response to Dave and Mostolog, each log line is a valid JSON and I would really prefer to only rely on liblognorm for parsing and not depend on mmjsonparse. Besides reluctance of installi

Re: [rsyslog] Send logs in JSON format to Elasticsearch

2017-02-23 Thread David Lang
On Thu, 23 Feb 2017, Alec Swan via rsyslog wrote: Liblognorm v2 supports json parser but it looks like it cannot be applied to the entire log message, which is sad. sure it can. you can configure mmnormalize to pass $msg to the parser engine, $rawmsg, or $!whatever. If your messages arrive