Re: [rsyslog] foreach in json array got segment fault?

2015-04-14 Thread singh.janmejay
It actually wasn't an omelasticsearch issue, the problem was related to actions and call statements not being initialized correctly when inside foreach body. It'd have failed with omfile as well, when used as an async-action. BTW, omelasticsearch has more file-grained error-reporting controls

Re: [rsyslog] foreach in json array got segment fault?

2015-04-14 Thread chenlin rao
Great! The problem is fixed! BTW: the `omelasticsearch` would cause segmentfault when it got error response and errorfile option no define and not running with valgrind. I think we might change this because errorfile record all bulk body including the right ones. Then the file size grow so

Re: [rsyslog] foreach in json array got segment fault?

2015-04-14 Thread singh.janmejay
The error-file related problem seems to be a different issue. Let me try to reproduce that. In the meanwhile, can you please share the backtraces from core-dump? (with debug symbols). On Tue, Apr 14, 2015 at 3:22 PM, chenlin rao rao.chen...@gmail.com wrote: The segmentfault I told in my last

Re: [rsyslog] foreach in json array got segment fault?

2015-04-14 Thread chenlin rao
The segmentfault I told in my last mail was happened after I use your master code. Means: * omes with errorfile would die in rsyslog-v8.8 * omes with errorfile run ok in rsyslog-your-fork. * omes without errorfile die in rsyslog-your-fork. Thank you for your comment about error file control.

Re: [rsyslog] rsyslog regex bre or operator

2015-04-14 Thread gaelor couilleaux
Sorry for the mistake, i'm using rsyslogd 5.10 ( can't upgrade because using omoracle module): rsyslogd 5.10.0, compiled with:FEATURE_REGEXP: YesFEATURE_LARGEFILE: NoGSSAPI Kerberos 5 support: No

Re: [rsyslog] foreach in json array got segment fault?

2015-04-14 Thread singh.janmejay
Yep, that is precisely the race. When copyMsg is turned on for that action, that race is not supposed to happen, I need to look deeper into this failure though. -- Regards, Janmejay PS: Please blame the typos in this mail on my phone's uncivilized soft keyboard sporting it's not-so-smart-assist

Re: [rsyslog] foreach in json array got segment fault?

2015-04-14 Thread Rainer Gerhards
2015-04-14 15:21 GMT+02:00 singh.janmejay singh.janme...@gmail.com: Yep, that is precisely the race. When copyMsg is turned on for that action, that race is not supposed to happen, I need to look deeper into this failure though. This may actually be an unrelated problem, probably related to

Re: [rsyslog] Best practice for an application to get structured data to rsyslog

2015-04-14 Thread singh.janmejay
Have you looked at mmjsonparse? It solves the problem of de-serializing structured-messages handed-over to rsyslog in JSON-serialized form. For dual-mode: structured and unstructured, 2 common approaches exist. - Passing structured messages as JSON and optionally handling the differently on

Re: [rsyslog] foreach in json array got segment fault?

2015-04-14 Thread singh.janmejay
I suspect that too. Will look at it today. -- Regards, Janmejay PS: Please blame the typos in this mail on my phone's uncivilized soft keyboard sporting it's not-so-smart-assist technology. On Apr 14, 2015 10:30 PM, Rainer Gerhards rgerha...@hq.adiscon.com wrote: 2015-04-14 15:21 GMT+02:00

Re: [rsyslog] foreach in json array got segment fault?

2015-04-14 Thread chenlin rao
So much thanks to you. It's totally OK now! 2015-04-15 10:37 GMT+08:00 singh.janmejay singh.janme...@gmail.com: There was an uninitialized pointer (the backtrace you posted was trying to free it). Can you test with latest 'master' on my fork again? On Wed, Apr 15, 2015 at 5:17 AM,

[rsyslog] Best practice for an application to get structured data to rsyslog

2015-04-14 Thread Ezell, Matthew A.
Hello- What is the current best practice for a portable application to get structured data to rsyslog? Most modern syslog daemons seem to support some type of JSON format, but applications still tend to use the old syslog(3) function for logging. If an application emits CEE JSON directly to

Re: [rsyslog] foreach in json array got segment fault?

2015-04-14 Thread singh.janmejay
No problem, my documentation work is done too, will create the PR. You probably want to merge the PR over 8.9.0 for local use. On Wed, Apr 15, 2015 at 8:49 AM, chenlin rao rao.chen...@gmail.com wrote: So much thanks to you. It's totally OK now! 2015-04-15 10:37 GMT+08:00 singh.janmejay

Re: [rsyslog] foreach in json array got segment fault?

2015-04-14 Thread singh.janmejay
There was an uninitialized pointer (the backtrace you posted was trying to free it). Can you test with latest 'master' on my fork again? On Wed, Apr 15, 2015 at 5:17 AM, singh.janmejay singh.janme...@gmail.com wrote: I suspect that too. Will look at it today. -- Regards, Janmejay PS:

Re: [rsyslog] Best practice for an application to get structured data to rsyslog

2015-04-14 Thread Ezell, Matthew A.
Sure, as a system administrator it's pretty clear how best to handle this. If there's CEE JSON data coming over the wire, use mmjsonparse. If it's unstructured traditional syslog(3) data, use mmnormalize to try to extract relevant fields based on rules I setup. Write the traditional message

Re: [rsyslog] Best practice for an application to get structured data to rsyslog

2015-04-14 Thread David Lang
On Wed, 15 Apr 2015, Ezell, Matthew A. wrote: Hello- What is the current best practice for a portable application to get structured data to rsyslog? Most modern syslog daemons seem to support some type of JSON format, but applications still tend to use the old syslog(3) function for logging.

Re: [rsyslog] Best practice for an application to get structured data to rsyslog

2015-04-14 Thread David Lang
On Wed, 15 Apr 2015, Ezell, Matthew A. wrote: Sure, as a system administrator it's pretty clear how best to handle this. If there's CEE JSON data coming over the wire, use mmjsonparse. If it's unstructured traditional syslog(3) data, use mmnormalize to try to extract relevant fields based on

Re: [rsyslog] foreach in json array got segment fault?

2015-04-14 Thread Rainer Gerhards
2015-04-14 14:43 GMT+02:00 David Lang da...@lang.hm: On Tue, 14 Apr 2015, chenlin rao wrote: FYI: The pstats show `mainQ.enqueued: 1322, action_json2log-es1003.enqueued: 6684` with interval 60s. Not a so large flow. If i'm understanding what's happening from reading this thread, when the

Re: [rsyslog] rsyslog regex bre or operator

2015-04-14 Thread gaelor couilleaux
Hello, Do you have an exemple with if condition and a regex method ? I must write it like this : if $hostname !regex ^.*\.gemalto\.com$ or $hostname !regex ^dct.*$ then ~ ? THX in Advanced Date: Fri, 10 Apr 2015 05:14:48 -0700 From: da...@lang.hm To: rsyslog@lists.adiscon.com Subject: Re:

Re: [rsyslog] rsyslog regex bre or operator

2015-04-14 Thread Rainer Gerhards
2015-04-14 15:07 GMT+02:00 gaelor couilleaux gae...@hotmail.fr: I'm using: rsyslogd 7.6.1, compiled with:FEATURE_REGEXP: YesFEATURE_LARGEFILE: NoGSSAPI Kerberos 5 support: NoFEATURE_DEBUG (debug build,

Re: [rsyslog] foreach in json array got segment fault?

2015-04-14 Thread David Lang
On Tue, 14 Apr 2015, chenlin rao wrote: FYI: The pstats show `mainQ.enqueued: 1322, action_json2log-es1003.enqueued: 6684` with interval 60s. Not a so large flow. If i'm understanding what's happening from reading this thread, when the foreach loop is processing a single message into a

Re: [rsyslog] rsyslog regex bre or operator

2015-04-14 Thread gaelor couilleaux
I'm using: rsyslogd 7.6.1, compiled with:FEATURE_REGEXP: YesFEATURE_LARGEFILE: NoGSSAPI Kerberos 5 support: NoFEATURE_DEBUG (debug build, slow code): No 32bit Atomic operations supported: Yes