[rsyslog] Is some tools like sequence analyze in lognormalizer?

Hello, rsyslog-users: I just find an interesting project named sequence in https://github.com/zentures/sequence/tree/master/cmd/sequence. It can 'analyze a log file and output a list of patterns that will match all the log messages'. And its document said that it's similar as libnormal,

Re: [rsyslog] Are we building an ERK stack?

https://github.com/rsyslog/rsyslog/pull/1099 2016-11-25 16:46 GMT+08:00 mosto...@gmail.com <mosto...@gmail.com>: > Thanks! > > It's your mmdblookup opensourced? > > > El 25/11/16 a las 03:46, chenlin rao escribió: > > re-upload an english version. The content was a

Re: [rsyslog] Are we building an ERK stack?

-11-25 15:39 GMT+08:00 Rainer Gerhards <rgerha...@hq.adiscon.com>: > 2016-11-25 8:26 GMT+01:00 chenlin rao <rao.chen...@gmail.com>: > > - rewrite most of mmgrok into mmnormalize+rainerscript. Except PHP > slowlog > > only. We want to translate the memory ad

Re: [rsyslog] Are we building an ERK stack?

0% when we changed the default templates to C definitions. > > It's a very useful slide deck. How has the 5.x version of ES changed > things there. > > David Lang > > On Fri, 25 Nov 2016, chenlin rao wrote: > > Date: Fri, 25 Nov 2016 10:46:27 +0800 >> From: chenlin rao

Re: [rsyslog] Are we building an ERK stack?

re-upload an english version. The content was a little old though. 2016-11-23 22:39 GMT+08:00 mosto...@gmail.com : > > http://www.slideshare.net/chenryn/elk-stack-at-weibocom > > I NEED the english version :P > > ___ > rsyslog

Re: [rsyslog] Are we building an ERK stack?

ERK +1, I have published my experiement at http://www.slideshare.net/chenryn/elk-stack-at-weibocom rsyslog-imsock -> rsyslog-omfwd -> rsyslog-imptcp -> rsyslog-mmnormalize/rsyslog-mmgrok/rsyslog-mmdblookup/rsyslog-mmfields/rainerscripts... -> rsyslog-omkafka -> kafka -> hangout

Re: [rsyslog] Logstash vs. omelasticsearch

I remember I had sent some message at here or github? One year ago, I try to use queue.dequeueslowdown to force rsyslog send a larger es bulk, then find the disk queue would only read 8 messages per second, no matter how quickly the memory queue recovered. sorry I can't give more informations

Re: [rsyslog] Logstash vs. omelasticsearch

There is a logstash plugin named logstash-filter-de_dot. And because logstash has a force queue to send a larger bulk_size to ES which is better in ES recommandition, I suggest to use rsyslog-omelasticsearch only for not so high load -- but need to watch it as quickly-- use case. 2016-11-18

Re: [rsyslog] Can we have a minimum bulk size for omelasticsearch?

hello, everyone. Is there any process about how to force a larger batch? I remember there was an email that rainer said he would implement some options to queue.c in the future. 2015-08-31 16:06 GMT+08:00 Radu Gheorghe : > Hi David, > > This sounds interesting, I

Re: [rsyslog] Can rsyslog-imfile record line number into metadata?

imfile does not do this, but it seems like a reasonable thing > to > > add, can you file an enhancement request at > > https://github.com/rsyslog/rsyslog ? > > > > David Lang > > > > On Mon, 11 Jan 2016, chenlin rao wrote: > > > > Date: Mon, 11 Jan 2

[rsyslog] Can rsyslog-imfile record line number into metadata?

I know the rsyslog-imfile can now record the filename into metadata. How about record the line number into metadata too? just like the `FNR` in awk. There is a use case that we could search for a range query of FNR to implement splunk's "context" in elasticsearch.

Re: [rsyslog] Fwd: How do I get rid of inactive external module processes started by omprog

+1. omprog used to fork thousands of subprocess in rsyslog-v8(v7 is ok). After adding `forcesingleinstance`, most of those omprog actions(I have 400+ omprog actions in my rsyslog.conf) are OK now, but I can see a few actios still has two forks. 2015-12-06 5:16 GMT+08:00 Manoj Kumar

Re: [rsyslog] [RFC] Log-forward destination-cluster support

Is this suggest any progress? I find my rsyslog server behind LVS output many error logs like "rsyslogd-2165: netstream session 0x7ff978739ce0 from 172.16.140.13 will be closed due to error [v8.11.0 try http://www.rsyslog.com/e/2165 ]". If I set the omfwd target of clients to only one server,

Re: [rsyslog] RFC: dynamic-stats support

I hope there is a stats about metrics based on $programname, $severity, $fromhost-ip etc, extends the ruleset(impstats). 2015-10-07 16:19 GMT+08:00 singh.janmejay : > -- > Regards, > Janmejay > > PS: Please blame the typos in this mail on my phone's uncivilized soft >

[rsyslog] re_extract() fails in 8.12.0 and 8.13.0

Hello all: I try to upgrade from 8.11.0 to 8.13.0, but got configuration parse fail. I search the github issue and find that this fails already exist in 8.12.0: There is nearly one month away from the issue created, no comment about it. Would

Re: [rsyslog] recommendations for omelasticsearch queue sizes

dequeuebatchsize don't has the same meaning as logstash flush_size. You can check http://www.gossamer-threads.com/lists/rsyslog/users/17550 and http://www.gossamer-threads.com/lists/rsyslog/users/17825 for more informations. btw: use too many actions may got 429 error from elasticsearch. I use

Re: [rsyslog] Can we have a minimum bulk size for omelasticsearch?

I have another reason to support this idea. By now, we can use `queue.dequeueslowdown` to force a little larger bulk size for omelasticsearch queue. but when this run into DA queue, the consume of DA queue is consoled by the queue.dequeueslowdown options too. So, I saw my DA size decearse only 8

Re: [rsyslog] how to force a larger omelasticsearch bulk size?

-18 17:18 GMT+08:00 Rainer Gerhards rgerha...@hq.adiscon.com: 2015-06-18 5:13 GMT+02:00 chenlin rao rao.chen...@gmail.com: yes. you are right. FYI, This rsyslog server is sending a short msg with 100B size. And I use a special ES template with _source disabled for it. I check another

[rsyslog] substr functions in conf?

I saw `substr($fromhost-ip,0,5)` example in the lookup_table functions' offical document. But I can't find any documents about `substr` function self. When I use substr in my rsyslog.conf, the `rsyslogd -N1` testing command return OK, but the result string always be null(). So, is it a wrong

Re: [rsyslog] Can rainerscript return syslogTime?

And another datestring as [03/Aug/2015:14:04:43 +0800]. 2015-08-03 14:04 GMT+08:00 chenlin rao rao.chen...@gmail.com: I want convert date in php-errolog: [03-Aug-2015 04:54:54 Asia/Shanghai] PHP Fatal error: Or date in php-slowlog: [03-Aug-2015 14:02:14] [pool v5] pid 27490 into 2015

Re: [rsyslog] Can rainerscript return syslogTime?

from? David Lang On Mon, 3 Aug 2015, chenlin rao wrote: %b in strptime, means 'Aug' for now. 2015-08-03 13:19 GMT+08:00 David Lang da...@lang.hm: what format is %b? (apache logfile uses %b for bytes returned. it may be possible to parse the data with mmnormalize. you can also use

Re: [rsyslog] Can rainerscript return syslogTime?

Great news of the new version libnormal. By now, how mmnormailize read as a timestamp?I only find date-iso, time-24hr etc in manual, and I think these return strings in json too. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog

Re: [rsyslog] Can rainerscript return syslogTime?

It's difficult to transform %b into rfc339 format. So many logs use %b , error log, slow log etc. 2015-08-03 11:39 GMT+08:00 David Lang da...@lang.hm: On Mon, 3 Aug 2015, chenlin rao wrote: I want a function like logstash-filter-date. For example: ``` $!datetime = strptime(%d/%m/%Y:%H

[rsyslog] Can rainerscript return syslogTime?

I want a function like logstash-filter-date. For example: ``` $!datetime = strptime(%d/%m/%Y:%H:%M:%s, $!json!datetime); template(name=ls type=list) {property(name=!datetime dateFormat=rfc3339)...} ``` But seems like rainerscript can only return number, string, array or json?

Re: [rsyslog] Can rainerscript return syslogTime?

2015, chenlin rao wrote: It's difficult to transform %b into rfc339 format. So many logs use %b , error log, slow log etc. 2015-08-03 11:39 GMT+08:00 David Lang da...@lang.hm: On Mon, 3 Aug 2015, chenlin rao wrote: I want a function like logstash-filter-date. For example: ``` $!datetime

[rsyslog] How rsyslog-v8 imfile handle the logrotate files?

Usually, logrotate mv the logfile to a new name like logfile.2015.07.01 and send a HUP to process to create a new file named logfile. But the imfile in rsyslog-v8 use inotify to listen at the logfile.2015.07.01 because mv keep the inode. How to keep imfile listen file by name, or refresh the

Re: [rsyslog] how to force a larger omelasticsearch bulk size?

, and possibly a little dangerous too, but sizing that thread-pool is definitely easy. Just size it to your need and it'll shape the batch-size optimally when under pressure (like David explained). On Wed, Jun 17, 2015 at 6:14 PM, chenlin rao rao.chen...@gmail.com wrote: well, there is something I

Re: [rsyslog] how to force a larger omelasticsearch bulk size?

this helps. Best regards, Radu -- Performance Monitoring * Log Analytics * Search Analytics Solr Elasticsearch Support * http://sematext.com/ On Wed, Jun 17, 2015 at 6:23 AM, chenlin rao rao.chen...@gmail.com wrote: So how can I define the output queue configuration? I

Re: [rsyslog] how to force a larger omelasticsearch bulk size?

10. Too small. Sometimes when I restart rsyslogd, the Content-Length grows to 8MB. Why~~ 2015-05-06 1:39 GMT+08:00 David Lang da...@lang.hm: On Tue, 5 May 2015, chenlin rao wrote: I'm using rsyslog-elasticsearch to writing nginx accesslog into Elasticsearch cluster. I found the document told

Re: [rsyslog] can field() return an array instread of string?

try to compile the liblognorm from git master, but got /lib64 file not recognized: is a directory error. When can liblognorm-1.1.2.rpm publish? 2015-05-29 4:00 GMT+08:00 David Lang da...@lang.hm: On Thu, 28 May 2015, chenlin rao wrote: The url params string, like 'trim_level=1uicode

[rsyslog] v8 omprog can't limit the num of sub process?

I use omprog in rsyslog-v7.6 for a long time. Each omprog would run only one sub process. But when I upgrade to rsyslog-v8, I found the omprog might fork too many sub processes. Thousands of sub processes that my server could be down Has anyone face such case?

[rsyslog] can field() return an array instread of string?

Hi all: I want to split my `$.urlargs` into a hash like `{k1:v1,k2:v2}`, but field() function only return the matchnbr substring, how can I got the whole array of kv? ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog

Re: [rsyslog] can field() return an array instread of string?

=1001with_common_cmt=1filter_by_author=0'. I want some functions just like logstash/filters/kv. Or split such string into an array by '', then I use `foreach` to process the array? 2015-05-28 17:33 GMT+08:00 David Lang da...@lang.hm: On Thu, 28 May 2015, chenlin rao wrote: Hi all: I want

[rsyslog] how to remove the blank space in $!all-json string?

Hi all. Can we remove the blank space inside the json strings? I like to use mmjsonparse and omelasticsearch plugins. But the json strings contain too many blank space, so we need more net flow, and elasticsearch would store the blank space bit in it's _source JSON which means more disk

Re: [rsyslog] how to remove the blank space in $!all-json string?

Rainer Gerhards rgerha...@hq.adiscon.com: sample please. But I need to say that $!all-json is pretty fixed... 2015-05-25 11:31 GMT+02:00 chenlin rao rao.chen...@gmail.com: Hi all. Can we remove the blank space inside the json strings? I like to use mmjsonparse and omelasticsearch

Re: [rsyslog] how to remove the blank space in $!all-json string?

No, ES store the raw JSON in _source field. We can set `_size: { enabled: true }` to check the record size. $ curl 10.19.0.97:9200/testindex/testtype/AU2OSfj0ZRvQT5qcC_l3?fields=_size,_source

Re: [rsyslog] cee json + mmsequence

mmsequence is deprecated? So which plugin instead? 2015-05-22 21:15 GMT+08:00 singh.janmejay singh.janme...@gmail.com: Cool, thats the answer I was looking for. On Fri, May 22, 2015 at 5:55 PM, Rainer Gerhards rgerha...@hq.adiscon.com wrote: 2015-05-21 17:40 GMT+02:00 singh.janmejay

[rsyslog] how to force a larger omelasticsearch bulk size?

I'm using rsyslog-elasticsearch to writing nginx accesslog into Elasticsearch cluster. I found the document told that the plugin would use queue.dequeuesize as the bulk size.But my tcpdump show that every POST only has 8-9 events in the bulk body while my input flow is nearly 10k per second. How

Re: [rsyslog] foreach in json array got segment fault?

0x003f6f4079d1 in start_thread () from /lib64/libpthread.so.0 #17 0x003f6f0e886d in clone () from /lib64/libc.so.6 Line numbers for some frames are slightly different, but it seems to be the same failure. On Thu, Apr 9, 2015 at 5:10 PM, chenlin rao rao.chen...@gmail.com wrote: I

Re: [rsyslog] foreach in json array got segment fault?

may want to check it. I'll post a PR to rsyslog/master once I finish the documentation enhancement, until next release you probably would want to merge that PR on top of 8.8.0 (or 8.9.0) tag. I'll update this thread once PR is ready On Tue, Apr 14, 2015 at 2:43 PM, chenlin rao rao.chen

Re: [rsyslog] foreach in json array got segment fault?

So much thanks to you. It's totally OK now! 2015-04-15 10:37 GMT+08:00 singh.janmejay singh.janme...@gmail.com: There was an uninitialized pointer (the backtrace you posted was trying to free it). Can you test with latest 'master' on my fork again? On Wed, Apr 15, 2015 at 5:17 AM,

Re: [rsyslog] foreach in json array got segment fault?

phone's uncivilized soft keyboard sporting it's not-so-smart-assist technology. On Apr 9, 2015 3:20 PM, chenlin rao rao.chen...@gmail.com wrote: janmyjay: I don't think that's a json-c problem, because if I use omfile inside foreach, rsyslogd never died. I also try to compile

Re: [rsyslog] foreach in json array got segment fault?

. While this fixes the value not being an array problem, you still want to move to sync mmjsonparse call. On Mon, Apr 6, 2015 at 6:07 PM, chenlin rao rao.chen...@gmail.com wrote: yes. rsyslogd 8.8.0.ad1, compiled with: PLATFORM: x86_64-redhat-linux-gnu PLATFORM (lsb_release -d

Re: [rsyslog] foreach in json array got segment fault?

phone's uncivilized soft keyboard sporting it's not-so-smart-assist technology. On Apr 9, 2015 3:20 PM, chenlin rao rao.chen...@gmail.com wrote: janmyjay: I don't think that's a json-c problem, because if I use omfile inside foreach, rsyslogd never died. I also try to compile

Re: [rsyslog] action queues vs message modifier modules vs dequeuebatchsize

rainer: What's main queue dequeuebatchsize meaning? I should define the options in `action()`, `ruleset()` or `$MainMsgQueuedequeuebatchsize`? I'm not so clear about ruleset queue and action queue. Especially one time, I defined `queue.workerthreads=5` both in my two different rulesets,

Re: [rsyslog] foreach in json array got segment fault?

default Runtime Instrumentation (slow code): No uuid support: Yes Number of Bits in RainerScript integers: 64 2015-04-06 20:32 GMT+08:00 singh.janmejay singh.janme...@gmail.com: So it is 8.8.0? On Mon, Apr 6, 2015 at 5:33 PM, chenlin rao rao.chen...@gmail.com wrote: I install latest version

Re: [rsyslog] foreach in json array got segment fault?

BTW: I try to re-run such cofiguration at CentOS5，but rsyslog v8-el5 told me `foreach` is not supportted in future. I didn't see such warning in rsyslog v8-el6? 2015-04-02 14:54 GMT+08:00 chenlin rao rao.chen...@gmail.com: I check the `}]` end just because there are some too long lines( 20MB

Re: [rsyslog] foreach in json array got segment fault?

I check the `}]` end just because there are some too long lines( 20MB+) might be transcated. Those line would cause mmjsonparse crash. I think this is also a place we can improve, but it's another question. 2015-04-02 14:48 GMT+08:00 chenlin rao rao.chen...@gmail.com: @cee:{msg:[{content:Error

Re: [rsyslog] foreach in json array got segment fault?

FAIL: json_array_looping.sh Rainer 2015-03-21 17:50 GMT+01:00 chenlin rao rao.chen...@gmail.com: thanks for the information. But rsyslogd also fault after I change mmjsonparse action config as `action ( type=mmjsonparse name=action_jsonarray-parse`. output as follow: 6008.997308131

Re: [rsyslog] foreach in json array got segment fault?

sense to mmjsonparse/mmfields etc? 2015-03-22 0:07 GMT+08:00 Rainer Gerhards rgerha...@hq.adiscon.com: 2015-03-21 14:50 GMT+01:00 chenlin rao rao.chen...@gmail.com: $MaxMessageSize 32m module( load=imtcp ) module( load=imuxsock ) module( load=imklog ) module( load=mmfields ) module( load

Re: [rsyslog] foreach in json array got segment fault?

, chenlin rao rao.chen...@gmail.com wrote: I try to build rsyslogd from github master with ./configure --enable-debug --enable-valgrind --enable-memcheck --enable-elasticsearch --enable-mmjsonparse --enable-mmsequence --enable-mmfields --disable-liblogging-stdlog --enable-omruleset

Re: [rsyslog] foreach in json array got segment fault?

btw: if I change omelasticsearch/omfwd to omfile, rsyslogd would be fine... 2015-03-20 20:13 GMT+08:00 chenlin rao rao.chen...@gmail.com: 3498.767218405:main Q[DA]:Reg/w0: rainerscript: var 200:!msg: '[ { uid: 1941604034, request_header: {\Accept-Encoding\:\gzip,deflate\}, network_type: wifi

Re: [rsyslog] foreach in json array got segment fault?

singh.janmejay singh.janme...@gmail.com: Can you please build with debug symbols and repeat the valgrind run? -- Regards, Janmejay PS: Please blame the typos in this mail on my phone's uncivilized soft keyboard sporting it's not-so-smart-assist technology. On Mar 20, 2015 6:03 PM, chenlin rao

Re: [rsyslog] foreach in json array got segment fault?

not-so-smart-assist technology. On Mar 19, 2015 11:10 PM, chenlin rao rao.chen...@gmail.com wrote: Hello everyone. I just learnt a foreach syntax from `src/tests/json_array_looping.sh`, so I try to parse my logdata(yes, long json array in msg) as follow: ``` $MaxMessageSize 256k

[rsyslog] foreach in json array got segment fault?

Hello everyone. I just learnt a foreach syntax from `src/tests/json_array_looping.sh`, so I try to parse my logdata(yes, long json array in msg) as follow: ``` $MaxMessageSize 256k template( name=local6JsonArray type=string string=%$.line%\n ) Ruleset( name=forwardRuleSetJsonArray ) {

[rsyslog] is there any output module using epoll?

I found omfwd would discard msg if I pass more than 1000msg per second. So I need to use mmsequence to split intto several omfwd queues. While the input modules can process lots more message than omfwd, and I found the imudp/imptcp use epoll, is there any output module using epoll?

[rsyslog] help for sudden buffer size

I have a 100 servers using rsyslog v7 sending messages to a VIP which has 10 really rsyslog servers. And I use impstats to monitor them. every node has 300k msgs enqueued per 5 minutes. But some node, 10 or so, may has 10~200k msgs size per 5 minutes! If I restart rsyslogd on one node, it would be

[rsyslog] Why mainQ store messages in it's memory size?

I use rsyslog v7, and has several actions using omfwd or omprog. But today I found the output data flow decrease quickly. I check my `impstats_log`, and found that every action pstats (no failed, no size, no discarded, no suspended) is well, but mainQ always records as follow: {name:main

Re: [rsyslog] Why mainQ store messages in it's memory size?

.* /var/log/cron uucp,news.crit /var/log/spooler 2014-11-03 20:08 GMT+08:00 David Lang da...@lang.hm: On Mon, 3 Nov 2014, chenlin rao wrote: I use rsyslog v7, and has several actions using omfwd or omprog. But today I found the output data