Thanks from me as well. Will merge in the not so distant future. I'd
like to have a deeper look at the changes and as we are weeks away
from next release, it probably doesn't hurt to keep the PR a couple of
days open (while I am deeply inside lognorm ;)).
Rainer
2015-04-15 5:21 GMT+02:00
Sounds good.
Realized I haven't posted PR link here yet.
PR: https://github.com/rsyslog/rsyslog/pull/303
On Wed, Apr 15, 2015 at 2:55 PM, Rainer Gerhards
rgerha...@hq.adiscon.com wrote:
Thanks from me as well. Will merge in the not so distant future. I'd
like to have a deeper look at the
It actually wasn't an omelasticsearch issue, the problem was related
to actions and call statements not being initialized correctly when
inside foreach body.
It'd have failed with omfile as well, when used as an async-action.
BTW, omelasticsearch has more file-grained error-reporting controls
Great! The problem is fixed!
BTW: the `omelasticsearch` would cause segmentfault when it got error
response and errorfile option no define and not running with valgrind.
I think we might change this because errorfile record all bulk body
including the right ones. Then the file size grow so
The error-file related problem seems to be a different issue. Let me
try to reproduce that.
In the meanwhile, can you please share the backtraces from core-dump?
(with debug symbols).
On Tue, Apr 14, 2015 at 3:22 PM, chenlin rao rao.chen...@gmail.com wrote:
The segmentfault I told in my last
The segmentfault I told in my last mail was happened after I use your
master code. Means:
* omes with errorfile would die in rsyslog-v8.8
* omes with errorfile run ok in rsyslog-your-fork.
* omes without errorfile die in rsyslog-your-fork.
Thank you for your comment about error file control.
Yep, that is precisely the race.
When copyMsg is turned on for that action, that race is not supposed to
happen, I need to look deeper into this failure though.
--
Regards,
Janmejay
PS: Please blame the typos in this mail on my phone's uncivilized soft
keyboard sporting it's not-so-smart-assist
2015-04-14 15:21 GMT+02:00 singh.janmejay singh.janme...@gmail.com:
Yep, that is precisely the race.
When copyMsg is turned on for that action, that race is not supposed to
happen, I need to look deeper into this failure though.
This may actually be an unrelated problem, probably related to
I suspect that too. Will look at it today.
--
Regards,
Janmejay
PS: Please blame the typos in this mail on my phone's uncivilized soft
keyboard sporting it's not-so-smart-assist technology.
On Apr 14, 2015 10:30 PM, Rainer Gerhards rgerha...@hq.adiscon.com
wrote:
2015-04-14 15:21 GMT+02:00
So much thanks to you. It's totally OK now!
2015-04-15 10:37 GMT+08:00 singh.janmejay singh.janme...@gmail.com:
There was an uninitialized pointer (the backtrace you posted was
trying to free it).
Can you test with latest 'master' on my fork again?
On Wed, Apr 15, 2015 at 5:17 AM,
No problem, my documentation work is done too, will create the PR.
You probably want to merge the PR over 8.9.0 for local use.
On Wed, Apr 15, 2015 at 8:49 AM, chenlin rao rao.chen...@gmail.com wrote:
So much thanks to you. It's totally OK now!
2015-04-15 10:37 GMT+08:00 singh.janmejay
There was an uninitialized pointer (the backtrace you posted was
trying to free it).
Can you test with latest 'master' on my fork again?
On Wed, Apr 15, 2015 at 5:17 AM, singh.janmejay
singh.janme...@gmail.com wrote:
I suspect that too. Will look at it today.
--
Regards,
Janmejay
PS:
2015-04-14 14:43 GMT+02:00 David Lang da...@lang.hm:
On Tue, 14 Apr 2015, chenlin rao wrote:
FYI: The pstats show `mainQ.enqueued: 1322,
action_json2log-es1003.enqueued: 6684` with interval 60s. Not a so large
flow.
If i'm understanding what's happening from reading this thread, when the
On Tue, 14 Apr 2015, chenlin rao wrote:
FYI: The pstats show `mainQ.enqueued: 1322,
action_json2log-es1003.enqueued: 6684` with interval 60s. Not a so large
flow.
If i'm understanding what's happening from reading this thread, when the foreach
loop is processing a single message into a
Chenlin, can you please try again with 'master' from my fork?
Also, please add the following param to omelasticsearch action invocation:
--
action.copyMsg=on
--
I'll create a pull request(need to do some documentation change to
cover code changes) once you confirm the problem is fixed.
On Mon,
I had already modify my config to:
```
action ( type=mmjsonparse
name=action_jsonarray-parse
)
if ( re_match($msg, ']}$') ) then {
foreach ($.line in $!msg) do {
action (...
```
just after you told me async queue no help with mm**.
So, the failure was
Can you reproduce the failure with synchronously calling mmjsonparse?
I'll try to reproduce it again.
--
Regards,
Janmejay
PS: Please blame the typos in this mail on my phone's uncivilized soft
keyboard sporting it's not-so-smart-assist technology.
On Apr 9, 2015 3:20 PM, chenlin rao
janmyjay:
I don't think that's a json-c problem, because if I use omfile inside
foreach, rsyslogd never died.
I also try to compile and run your repo, process fault again and the
valgrind output keep totally same.
2015-04-08 21:26 GMT+08:00 singh.janmejay singh.janme...@gmail.com:
I had already modify my config to:
```
action ( type=mmjsonparse
name=action_jsonarray-parse
)
if ( re_match($msg, ']}$') ) then {
foreach ($.line in $!msg) do {
action (...
```
just after you told me async queue no help with mm**.
So, the failure was
I'll try to reproduce the failure with sync call to mm call. It may take
some time before I can look at it again though.
--
Regards,
Janmejay
PS: Please blame the typos in this mail on my phone's uncivilized soft
keyboard sporting it's not-so-smart-assist technology.
On Apr 9, 2015 5:07 PM,
BTW, here is the gdb-session-trace if you are interested:
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x71657700 (LWP 14762)]
0x7718c23b in array_list_get_idx () from
/lib/x86_64-linux-gnu/libjson-c.so.2
(gdb) bt
#0 0x7718c23b in
Spent some time on this today.
I had the crash reproducing consistently with tcpflood -C 1000 ...
Here is what I observed:
- When large number of messages (copy of the message you shared) are
sent, some messages get parsed with key 'msg' pointing to a
json-string (its contents is the entire
yes.
rsyslogd 8.8.0.ad1, compiled with:
PLATFORM: x86_64-redhat-linux-gnu
PLATFORM (lsb_release -d):
FEATURE_REGEXP: Yes
GSSAPI Kerberos 5 support: No
FEATURE_DEBUG (debug build, slow code): No
32bit Atomic operations supported: Yes
64bit Atomic operations supported: Yes
memory allocator: system
So it is 8.8.0?
On Mon, Apr 6, 2015 at 5:33 PM, chenlin rao rao.chen...@gmail.com wrote:
I install latest version from rsyslog_v8 yum repo in CentOS6.5.
And for more options in valgrind report, I compiled another version from
git master, ran into the same problem.
2015-04-06 13:55 GMT+08:00
It was looking very similar to the issue that Thomas reported on
http://www.gossamer-threads.com/lists/rsyslog/users/16204?do=post_view_threaded
To double-check, I cherry-picked the two patches in that PR, which
fixed the failing json_array_looping test.
So I'll now focus on Chenlin's problem.
BTW: I try to re-run such cofiguration at CentOS5，but rsyslog v8-el5 told
me `foreach` is not supportted in future. I didn't see such warning in
rsyslog v8-el6?
2015-04-02 14:54 GMT+08:00 chenlin rao rao.chen...@gmail.com:
I check the `}]` end just because there are some too long lines( 20MB+)
I
2015-04-02 7:35 GMT+02:00 singh.janmejay singh.janme...@gmail.com:
Hadn't noticed the use of msg key early-on, so ]} shouldn't be a
problem, but I can't reproduce it with input i crafted to work with
it.
Here is the crafted input:
@cee:{msg : [G\foo\:\10\}, G\bar\:\20\]}
This is what
I check the `}]` end just because there are some too long lines( 20MB+)
might be transcated. Those line would cause mmjsonparse crash. I think this
is also a place we can improve, but it's another question.
2015-04-02 14:48 GMT+08:00 chenlin rao rao.chen...@gmail.com:
@cee:{msg:[{content:Error
@cee:{msg:[{content:Error Domain=WBBussinessErrorDomain Code=20030
\the user base info is not exists!\ UserInfo=0x17127cc00
{NSLocalizedDescription=the user base info is not exists!,
WBBussinessErrorInfoKey=CFBasicHash 0x17127cb40 [0x193536c80]{type =
immutable dict, count = 4,\nentries =\n\t0 :
I couldn't spend too much time on it, but just saw it reproduce on my
local env (also running Ubuntu 12.04).
Will again get a chance to look at in a a few days (likely Monday).
Again, sorry for the delay.
On Thu, Apr 2, 2015 at 6:43 PM, singh.janmejay singh.janme...@gmail.com wrote:
Foreach
Foreach support is a recent addition.
On Thu, Apr 2, 2015 at 3:34 PM, singh.janmejay singh.janme...@gmail.com wrote:
I saw that mail, but by that time I had already setup the environment
to reproduce it with Chenlin's config.
I should pursue the CI failure first. Didn't know it was happening
Hi Chenlin,
I finally have some time to look at it. (Sorry for the delay.)
Can you please share a sample log-line? Feel free to anonymize it, but
please keep the structure and types as is.
I see that you are checking $msg ends with ]}, that means it is an
object, not an array.
If my
Hadn't noticed the use of msg key early-on, so ]} shouldn't be a
problem, but I can't reproduce it with input i crafted to work with
it.
Here is the crafted input:
@cee:{msg : [G\foo\:\10\}, G\bar\:\20\]}
This is what does fail it, im using a bad type (it doesn't do type-check yet):
@cee:{msg :
2015-03-30 19:55 GMT+02:00 singh.janmejay singh.janme...@gmail.com:
Sorry, I haven't had a chance to look at it yet.
Take your time, I know how challenging it is at times ;) I just wanted
to spread the good news that we have a good repro.
Rainer
Will get to this asap.
On Mon, Mar 30, 2015
Sorry, I haven't had a chance to look at it yet.
Will get to this asap.
On Mon, Mar 30, 2015 at 2:36 PM, Rainer Gerhards
rgerha...@hq.adiscon.com wrote:
Co-incidentally, an unrelated testbench run also experienced problems
in this area. I have attached the testbench log.
Here is the valgrind
thanks for the information. But rsyslogd also fault after I change
mmjsonparse action config as `action ( type=mmjsonparse
name=action_jsonarray-parse`. output as follow:
6008.997308131:main Q:Reg/w1 : eval expr 0x52f18f0, return datatype 'J'
6009.002050327:main Q:Reg/w1 : ACTION 16
2015-03-21 14:50 GMT+01:00 chenlin rao rao.chen...@gmail.com:
$MaxMessageSize 32m
module( load=imtcp )
module( load=imuxsock )
module( load=imklog )
module( load=mmfields )
module( load=mmjsonparse )
module( load=omelasticsearch )
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$MaxMessageSize 32m
module( load=imtcp )
module( load=imuxsock )
module( load=imklog )
module( load=mmfields )
module( load=mmjsonparse )
module( load=omelasticsearch )
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$RepeatedMsgReduction off
$WorkDirectory /data0/rsyslog
On Sun, 22 Mar 2015, chenlin rao wrote:
BTW: is it the workerthreads options make sense to mmjsonparse/mmfields
etc?
well, workerthreads = 30 is almost certinly wrong and would cause performance
problems
you should only increase workerthreads above the default of 1 if you have
measured
btw: if I change omelasticsearch/omfwd to omfile, rsyslogd would be fine...
2015-03-20 20:13 GMT+08:00 chenlin rao rao.chen...@gmail.com:
3498.767218405:main Q[DA]:Reg/w0: rainerscript: var 200:!msg: '[ { uid:
1941604034, request_header: {\Accept-Encoding\:\gzip,deflate\},
network_type: wifi,
Can you please build with debug symbols and repeat the valgrind run?
--
Regards,
Janmejay
PS: Please blame the typos in this mail on my phone's uncivilized soft
keyboard sporting it's not-so-smart-assist technology.
On Mar 20, 2015 6:03 PM, chenlin rao rao.chen...@gmail.com wrote:
btw: if I
I try to build rsyslogd from github master with ./configure --enable-debug
--enable-valgrind --enable-memcheck --enable-elasticsearch
--enable-mmjsonparse --enable-mmsequence --enable-mmfields
--disable-liblogging-stdlog --enable-omruleset
then process exit with:
7778.301430428:main
3498.767218405:main Q[DA]:Reg/w0: rainerscript: var 200:!msg: '[ { uid:
1941604034, request_header: {\Accept-Encoding\:\gzip,deflate\},
network_type: wifi, end_time: 1426836307406, dns_ip:
218.15.203.34,192.168.1.1, response_code: 200, response_data: {},
start_time: 1426836307328, act:
Can you please run this with valgrind and share its output on crash?
--
Regards,
Janmejay
PS: Please blame the typos in this mail on my phone's uncivilized soft
keyboard sporting it's not-so-smart-assist technology.
On Mar 19, 2015 11:10 PM, chenlin rao rao.chen...@gmail.com wrote:
Hello
Hello everyone.
I just learnt a foreach syntax from `src/tests/json_array_looping.sh`, so I
try to parse my logdata(yes, long json array in msg) as follow:
```
$MaxMessageSize 256k
template( name=local6JsonArray type=string string=%$.line%\n )
Ruleset( name=forwardRuleSetJsonArray )
{
45 matches
Mail list logo
