I believe that Janmejay suggests below (far below) that in the "second-last
branch in msgSetPropViaJSON" a ref count bump should be added before the call
to msgAddJSON.
I started with a freshly unpacked 8.10.0 tree and inserted the bump at that
location. That results in an rsyslogd that segfaults with my setup, although in
a different spot than before. This happens in both 8.9 and 8.10. My suggested
patch, which is in the body of msgAddJSON, only does the bump when name[1] is
'\0' (and from debugging this appears to be the case for $!).
If I am correct about what Janmejay is suggesting as a patch, then my patch
will bump the ref count for a subset of the cases in which his patch would bump
it. That is not to say that my patch is correct, just that further debugging is
needed by someone who is familiar with the guts of rsyslogd. (If I've
misinterpreted Janmejay's suggested fix, my apologies).
The backtrace for the new segfault is:
========
#0 0x00007ffff7192592 in json_object_object_to_json_string
(jso=0x7fffe8003150, pb=0x7fffe8006fd0, level=0, flags=1) at json_object.c:310
#1 0x00007ffff719248c in json_object_to_json_string_ext (jso=0x7fffe8003150,
flags=1) at json_object.c:276
#2 0x000000000041f005 in getJSONPropVal (pMsg=0x7ffff0004030, pProp=0x6c73b0,
pRes=0x7ffff4bdc8a0, buflen=0x7ffff4bdc89c,
pbMustBeFreed=0x7ffff4bdca56) at msg.c:2802
#3 0x0000000000420075 in MsgGetProp (pMsg=0x7ffff0004030, pTpe=0x6c7390,
pProp=0x6c73b0, pPropLen=0x7ffff4bdca44,
pbMustBeFreed=0x7ffff4bdca56, ttNow=0x7ffff4bdcb30) at msg.c:3285
#4 0x0000000000454a5a in tplToString (pTpl=0x6c50b0, pMsg=0x7ffff0004030,
iparam=0x7fffe8002d90, ttNow=0x7ffff4bdcb30)
at ../template.c:194
#5 0x000000000044e5c9 in prepareDoActionParams (pAction=0x6d7c30,
pWti=0x6ce940, pMsg=0x7ffff0004030, ttNow=0x7ffff4bdcb30)
at ../action.c:933
#6 0x000000000044f332 in processMsgMain (pAction=0x6d7c30, pWti=0x6ce940,
pMsg=0x7ffff0004030, ttNow=0x7ffff4bdcb30) at ../action.c:1300
#7 0x000000000044f979 in doSubmitToActionQ (pAction=0x6d7c30, pWti=0x6ce940,
pMsg=0x7ffff0004030) at ../action.c:1449
#8 0x0000000000445568 in execAct (stmt=0x6d7ba0, pMsg=0x7ffff0004030,
pWti=0x6ce940) at ruleset.c:202
#9 0x0000000000445f52 in scriptExec (root=0x6d41b0, pMsg=0x7ffff0004030,
pWti=0x6ce940) at ruleset.c:454
#10 0x000000000044619a in processBatch (pBatch=0x6ce978, pWti=0x6ce940) at
ruleset.c:508
#11 0x000000000045be5f in msgConsumer (notNeeded=0x0, pBatch=0x6ce978,
pWti=0x6ce940) at rsyslogd.c:575
...
========
At the site of the break we have a macro, json_object_object_foreachC. That
macro pulls the lh_table out of the json_object as json->o.c_object. For the
json_object being traversed in my rsyslogd, json->o.c_object is
0x21.Interestingly, most of the fields are 0x21, which is '!'. The macro treats
this as a pointer value and bombs.
I looked at the test code area and it looks to have a steep learning curve. The
code (rsyslogd conf, test client, and mmexternal target) I posted on github
(https://github.com/davidboles/snowflake.git) reproduces the problem quite
readily. This patch:
--- ../pristine/rsyslog-8.10.0/runtime/msg.c 2015-05-19 03:53:40.000000000
-0500
+++ runtime/msg.c 2015-06-09 14:47:51.066158522 -0500
@@ -4343,6 +4343,7 @@
}
if(name[1] == '\0') { /* full tree? */
+ json_object_get(json);
if(*pjroot == NULL)
*pjroot = json;
else
removes the issue and doesn't leak memory for my test case even when run
hundreds of thousand times.
I enabled testbench in my patched 8.10 tree and a make check ends with:
========
make check-TESTS
make[2]: Entering directory
`/home/dboles/scratch/rsyslog/build/rsyslog-8.10.0/tests'
make[3]: Entering directory
`/home/dboles/scratch/rsyslog/build/rsyslog-8.10.0/tests'
PASS: stop-localvar.sh
PASS: stop-msgvar.sh
PASS: fac_authpriv.sh
PASS: fac_local0.sh
PASS: fac_local7.sh
PASS: fac_mail.sh
PASS: fac_news.sh
PASS: fac_ftp.sh
PASS: fac_ntp.sh
PASS: fac_uucp.sh
PASS: fac_invld1.sh
PASS: fac_invld2.sh
PASS: fac_invld3.sh
PASS: fac_invld4_rfc5424.sh
PASS: rfc5424parser.sh
PASS: arrayqueue.sh
PASS: global_vars.sh
PASS: da-mainmsg-q.sh
PASS: validation-run.sh
PASS: empty-ruleset.sh
PASS: imtcp-multiport.sh
PASS: daqueue-persist.sh
PASS: diskqueue.sh
PASS: diskqueue-fsync.sh
make[3]: *** No rule to make target `rulesetmultiqueue.sh', needed by
`rulesetmultiqueue.sh.log'. Stop.
make[3]: Leaving directory
`/home/dboles/scratch/rsyslog/build/rsyslog-8.10.0/tests'
make[2]: *** [check-TESTS] Error 2
make[2]: Leaving directory
`/home/dboles/scratch/rsyslog/build/rsyslog-8.10.0/tests'
make[1]: *** [check-am] Error 2
make[1]: Leaving directory
`/home/dboles/scratch/rsyslog/build/rsyslog-8.10.0/tests'
make: *** [check-recursive] Error 1
========
I'd be happy to assist the rsyslog team in any way or answer any questions. For
now, I'll just proceed with my patched version.
David Boles
________________________________________
From: [email protected] [[email protected]] on
behalf of Rainer Gerhards [[email protected]]
Sent: Thursday, June 11, 2015 12:24 AM
To: rsyslog-users
Subject: Re: [rsyslog] Unable to use foreach
2015-06-11 7:15 GMT+02:00 singh.janmejay <[email protected]>:
> I haven't dug deep, but with 10 minutes worth of analysis, it seems
> like offender is: MsgSetPropsViaJSON
>
> This is what mmexternal calls, it gives it a string, which its
> supposed to parse as a json and then set properties if that json turns
> out to be a json-object (as opposed to array or string or
> numeric-litteral etc).
>
> Now the loop here(part of MsgSetPropsViaJSON):
> json_object_object_foreach(json, name, val) {
> msgSetPropViaJSON(pMsg, name, val, 0);
> }
>
> seems to be doing the 'setting' of values key-by-key by calling
> msgSetPropViaJSON.
>
> Convince yourself that the for-each macro doesn't increment ref-count
> of the value by peeking into json-c codebase (I looked at master as of
> Sep last year, haven't been keeping it up to date).
>
> Now look at the second-last branch in msgSetPropViaJSON, which calls
> msgAddJSON, but doesn't do a ref-count bump before that.
>
> Makes sense?
Thanks for looking into it. From the description it makes a lot of
sense, also this seems to occur only with mmexternal, so that looks
also like it points into the right direction.
>
> I am happy to fix it and create a PR for it, but it'll take me
> at-least a week (because I need to wrap up stuff before I can spend
> more time on this).
That's most probably faster than I can look at it.
Rainer
> In the mean-while, if anyone else wants to fix
> this please update this thread and then go about it. But should you
> choose to fix it, _please do write a test to cover the bug_, and run
> tests with valgrind before applying the fix to prove to yourself that
> it is indeed broken.
>
> On Thu, Jun 11, 2015 at 10:18 AM, singh.janmejay
> <[email protected]> wrote:
>> Once I have a little bit more time, I'll go through the thread and try
>> to help find the offender.
>>
>> On Thu, Jun 11, 2015 at 10:16 AM, singh.janmejay
>> <[email protected]> wrote:
>>> Ref-count increment is not required because callers are supposed to
>>> pass exclusive ownership while setting value of a key. What you are
>>> seeing is result of bad-usage (where caller probably free'ed up the
>>> json_object after setting it as value of some key).
>>>
>>> To understand this better, look at some call-sites. Let us take an
>>> example, say set <x> = <y>;
>>>
>>> It calls msgSetJSONFromVar from execSet. Notice how in each scenario
>>> the json_object passed to msgAddJSON is an exclusive copy.
>>>
>>> If you go through all the call-sites that invoke msgAddJSON, you'll
>>> see that exclusive-copy-only is an invariant that is always
>>> maintained.
>>>
>>> The "bad-usage" that I mentioned above means some caller has not
>>> maintained that invariant, and that lead to the json_object being
>>> free'ed after adding it to msg.
>>>
>>> Am I making sense?
>>>
>>> On Wed, Jun 10, 2015 at 2:39 AM, David Boles (dboles) <[email protected]>
>>> wrote:
>>>> I finished setting up a debugging environment for 8.10.0 and spent some
>>>> time stepping through the code. First, I noticed that the json_object that
>>>> was in pMsg got overwritten on an unrelated call to
>>>> json_object_new_object(). That suggested that the json-c library thought
>>>> it owned that memory while a pMsg still had a pointer to it. In reading
>>>> the code I noticed that in msgDelJSON() under the condition "if(name[1] ==
>>>> '\0')" there is a json_object_put() but under the matching condition in
>>>> msgAddJSON() there is no json_object_get().
>>>>
>>>> After more reading through the code to get a rough idea of code flow I
>>>> applied the following delta:
>>>>
>>>> --- ../pristine/rsyslog-8.10.0/runtime/msg.c 2015-05-19
>>>> 03:53:40.000000000 -0500
>>>> +++ runtime/msg.c 2015-06-09 14:47:51.066158522 -0500
>>>> @@ -4343,6 +4343,7 @@
>>>> }
>>>>
>>>> if(name[1] == '\0') { /* full tree? */
>>>> + json_object_get(json);
>>>> if(*pjroot == NULL)
>>>> *pjroot = json;
>>>> else
>>>>
>>>> and rebuilt. That change eliminated the overwriting of the pMsg->json
>>>> object and eliminated the segfault. I then performed the following steps:
>>>>
>>>> (1) launched the patched rsyslogd
>>>> (2) pumped 100K messages through it to reach steady state
>>>> (3) measured the memory footprint of rsyslogd
>>>> (4) pumped another 100K messages through it
>>>> (5) measured the RSS memory footprint of rsyslogd again and verified
>>>> that there was no change
>>>>
>>>> I don't know whether my patch is correct or not but I'll be continuing
>>>> down this path until I get feedback. If it is correct, then it's a fix for
>>>> issue #368.
>>>>
>>>> Thanks,
>>>>
>>>> David Boles
>>>> ________________________________________
>>>> From: David Boles (dboles)
>>>> Sent: Monday, June 08, 2015 4:36 PM
>>>> To: rsyslog-users
>>>> Subject: RE: [rsyslog] Unable to use foreach
>>>>
>>>> Note - I don't get a segfault when mmexternal is enabled unless I also
>>>> access $!. I can log the contents of my mmexternal rewritten "msg" without
>>>> error. However, because I'm logging "msg" I get a quote-ified
>>>> representation of the JSON that I want.
>>>>
>>>> Even if my Python script doesn't modify anything but "msg", as soon as I
>>>> either do the RSYSLOG_DebugFormat action (which accesses $!) or perform
>>>> either an omfile or ommongodb action with a template referencing $! then
>>>> rsyslogd segfaults. I don't know rsyslogd internals at all but below
>>>> you'll find a backtrace from the point of the segfault. I've confirmed
>>>> that it is getting the "!" property and am working to set up a debugging
>>>> environment now.
>>>>
>>>> This has been filed as an issue per Rainer's request several days ago.
>>>>
>>>> Thanks,
>>>>
>>>> David Boles
>>>>
>>>> ========
>>>>
>>>> Program received signal SIGSEGV, Segmentation fault.
>>>> [Switching to Thread 0x7ffff4a2e700 (LWP 4761)]
>>>> printbuf_reset (p=0xffffffffffffffff) at printbuf.c:182
>>>> 182 p->buf[0] = '\0';
>>>> Missing separate debuginfos, use: debuginfo-install
>>>> glib2-2.40.0-4.el7.x86_64 glibc-2.17-55.el7.x86_64 l
>>>> iblogging-1.0.5-1.el7.x86_64 libmongo-client-0.1.8-1.el7.x86_64
>>>> libuuid-2.23.2-22.el7_1.x86_64 systemd-l
>>>> ibs-208-20.el7_1.3.x86_64 zlib-1.2.7-13.el7.x86_64
>>>> (gdb) bt
>>>> #0 printbuf_reset (p=0xffffffffffffffff) at printbuf.c:182
>>>> #1 0x00007ffff719247e in json_object_to_json_string_ext
>>>> (jso=0x7fffe8002580, flags=flags@entry=1)
>>>> at json_object.c:274
>>>> #2 0x00007ffff71924da in json_object_to_json_string (jso=<optimized out>)
>>>> at json_object.c:286
>>>> #3 0x00007ffff7192d15 in json_object_get_string (jso=<optimized out>) at
>>>> json_object.c:738
>>>> #4 0x000000000041b7d1 in getJSONPropVal (pMsg=<optimized out>,
>>>> pProp=0x6aaa00,
>>>> pRes=pRes@entry=0x7ffff4a2d968, buflen=buflen@entry=0x7ffff4a2d95c,
>>>> pbMustBeFreed=0x7ffff4a2da42)
>>>> at msg.c:2802
>>>> #5 0x000000000041bbcc in MsgGetProp (pMsg=pMsg@entry=0x7ffff0004030,
>>>> pTpe=pTpe@entry=0x6aa9e0,
>>>> pProp=pProp@entry=0x6aaa00, pPropLen=pPropLen@entry=0x7ffff4a2da44,
>>>> pbMustBeFreed=pbMustBeFreed@entry=0x7ffff4a2da42,
>>>> ttNow=ttNow@entry=0x7ffff4a2db10) at msg.c:3285
>>>> #6 0x000000000044472c in tplToString (pTpl=0x6a8700,
>>>> pMsg=pMsg@entry=0x7ffff0004030,
>>>> iparam=iparam@entry=0x7fffe8002940, ttNow=ttNow@entry=0x7ffff4a2db10)
>>>> at ../template.c:194
>>>> #7 0x000000000043fbff in prepareDoActionParams (pWti=0x6b1fc0,
>>>> ttNow=0x7ffff4a2db10,
>>>> pMsg=0x7ffff0004030, pAction=0x6b9680) at ../action.c:933
>>>> #8 processMsgMain (pAction=pAction@entry=0x6b9680,
>>>> pWti=pWti@entry=0x6b1fc0, pMsg=0x7ffff0004030,
>>>> ttNow=ttNow@entry=0x7ffff4a2db10) at ../action.c:1300
>>>> #9 0x0000000000440337 in doSubmitToActionQ (pAction=0x6b9680,
>>>> pWti=0x6b1fc0, pMsg=<optimized out>)
>>>> at ../action.c:1449
>>>> #10 0x00000000004390a7 in execAct (stmt=0x6b95f0, pWti=0x6b1fc0,
>>>> pMsg=0x7ffff0004030) at ruleset.c:202
>>>> #11 scriptExec (root=<optimized out>, pMsg=pMsg@entry=0x7ffff0004030,
>>>> pWti=pWti@entry=0x6b1fc0)
>>>> at ruleset.c:454
>>>> #12 0x0000000000439884 in processBatch (pBatch=0x6b1ff8, pWti=0x6b1fc0) at
>>>> ruleset.c:508
>>>> #13 0x0000000000448cd4 in msgConsumer (notNeeded=<optimized out>,
>>>> pBatch=0x6b1ff8, pWti=0x6b1fc0)
>>>> at rsyslogd.c:575
>>>> #14 0x00000000004357ee in ConsumerReg (pThis=0x6b1b20, pWti=0x6b1fc0) at
>>>> queue.c:1897
>>>> #15 0x000000000043182e in wtiWorker (pThis=pThis@entry=0x6b1fc0) at
>>>> wti.c:334
>>>> #16 0x000000000043001a in wtpWorker (arg=0x6b1fc0) at wtp.c:389
>>>> #17 0x00007ffff79b1df3 in start_thread () from /lib64/libpthread.so.0
>>>> #18 0x00007ffff6aba3dd in clone () from /lib64/libc.so.6
>>>> (gdb) list
>>>> 177 }
>>>> 178 }
>>>> 179
>>>> 180 void printbuf_reset(struct printbuf *p)
>>>> 181 {
>>>> 182 p->buf[0] = '\0';
>>>> 183 p->bpos = 0;
>>>> 184 }
>>>> 185
>>>> 186 void printbuf_free(struct printbuf *p)
>>>> (gdb) p p
>>>> $1 = (struct printbuf *) 0xffffffffffffffff
>>>>
>>>>
>>>> ________________________________________
>>>> From: [email protected]
>>>> [[email protected]] on behalf of David Lang [[email protected]]
>>>> Sent: Monday, June 08, 2015 3:40 PM
>>>> To: rsyslog-users
>>>> Subject: Re: [rsyslog] Unable to use foreach
>>>>
>>>> given that input, what does your python script output?
>>>>
>>>> since you are getting a segfault when mmexternal is enabled, I would guess
>>>> that
>>>> you are returning something invalid and rsyslog is not properly handling
>>>> the
>>>> error.
>>>>
>>>> I'll also point out that what you are passing to your script is much
>>>> larger than
>>>> it needs to be (you have the same data in the original message, in
>>>> rawmessge, in
>>>> $! and sometimes in other things). I would suggest creating a template
>>>> that just
>>>> has $! in it, and set variables for anything you need that's not already in
>>>> there.
>>>>
>>>> David Lang
>>>>
>>>> On Mon, 8 Jun 2015, David Boles (dboles) wrote:
>>>>
>>>>> Date: Mon, 8 Jun 2015 20:34:45 +0000
>>>>> From: "David Boles (dboles)" <[email protected]>
>>>>> Reply-To: rsyslog-users <[email protected]>
>>>>> To: rsyslog-users <[email protected]>
>>>>> Subject: Re: [rsyslog] Unable to use foreach
>>>>>
>>>>> David,
>>>>>
>>>>> Sorry for the delay in responding - my focus was elsewhere temporarily.
>>>>> What you say about $! and new content below makes sense. I've tried an
>>>>> experiment with the RSYSLOG_DebugFormat mechanism by running with the
>>>>> following /etc/rsyslog.conf tweaked in three ways:
>>>>>
>>>>> --------
>>>>> module(load="imuxsock")
>>>>> module(load="mmexternal")
>>>>> module(load="mmjsonparse")
>>>>> module(load="ommongodb")
>>>>>
>>>>> template(name="yyd-log-info" type="list") {
>>>>> property(name="timestamp")
>>>>> constant(value=" ")
>>>>> property(name="hostname")
>>>>> constant(value=" ")
>>>>> property(name="msg")
>>>>> constant(value="\n")
>>>>> }
>>>>>
>>>>> # 1
>>>>> *.* /var/log/debug_pre.log;RSYSLOG_DebugFormat
>>>>>
>>>>> if ($fromhost-ip == '127.0.0.1' and $syslogfacility-text != 'kern') then {
>>>>> action(type="mmjsonparse")
>>>>> if $parsesuccess == "OK" then {
>>>>> # 2
>>>>> action(type="mmexternal"
>>>>> binary="/tmp/snowflake/rewriter.py"
>>>>> interface.input="fulljson" )
>>>>> action(type="omfile"
>>>>> template="yyd-log-info"
>>>>> file="/var/log/yyd_anon.log")
>>>>> }
>>>>> }
>>>>>
>>>>> # 3
>>>>> #*.* /var/log/debug_post.log;RSYSLOG_DebugFormat
>>>>> --------
>>>>>
>>>>> (1) One mode was with the mmexternal action after # 2 commented out and
>>>>> the DebugFormat action at # 3, yielding:
>>>>> --------
>>>>> Debug line with all properties:
>>>>> FROMHOST: 'dboles-victim-0', fromhost-ip: '127.0.0.1', HOSTNAME:
>>>>> 'dboles-victim-0', PRI: 12,
>>>>> syslogtag ':', programname: '', APP-NAME: '', PROCID: '-', MSGID: '-',
>>>>> TIMESTAMP: 'Jun 8 15:37:24', STRUCTURED-DATA: '-',
>>>>> msg: ' @cee:{"msg":"[umberlog test] testing ... 1, 2,
>>>>> 3\n","count":"2","el.0.type":"ladybug","el.0.name":"natalie","el.0.spots":"12","el.1.type":"ant","el.1.name":"B289","el.1.role":"worker","el.1.age":"9","pid":"32294","facility":"user","priority":"warn","uid":"1002","gid":"1002","host":"dboles-victim-0","program":"","timestamp":"2015-06-08T15:37:24.449217401-0400"}'
>>>>> escaped msg: ' @cee:{"msg":"[umberlog test] testing ... 1, 2,
>>>>> 3\n","count":"2","el.0.type":"ladybug","el.0.name":"natalie","el.0.spots":"12","el.1.type":"ant","el.1.name":"B289","el.1.role":"worker","el.1.age":"9","pid":"32294","facility":"user","priority":"warn","uid":"1002","gid":"1002","host":"dboles-victim-0","program":"","timestamp":"2015-06-08T15:37:24.449217401-0400"}'
>>>>> inputname: imuxsock rawmsg: '<12>Jun 8 15:37:24 : @cee:{"msg":"[umberlog
>>>>> test] testing ... 1, 2,
>>>>> 3\n","count":"2","el.0.type":"ladybug","el.0.name":"natalie","el.0.spots":"12","el.1.type":"ant","el.1.name":"B289","el.1.role":"worker","el.1.age":"9","pid":"32294","facility":"user","priority":"warn","uid":"1002","gid":"1002","host":"dboles-victim-0","program":"","timestamp":"2015-06-08T15:37:24.449217401-0400"}'
>>>>> $!:{ "msg": "[umberlog test] testing ... 1, 2, 3\n", "count": "2",
>>>>> "el.0.type": "ladybug", "el.0.name": "natalie", "el.0.spots": "12",
>>>>> "el.1.type": "ant", "el.1.name": "B289", "el.1.role": "worker",
>>>>> "el.1.age": "9", "pid": "32294", "facility": "user", "priority": "warn",
>>>>> "uid": "1002", "gid": "1002", "host": "dboles-victim-0", "program": "",
>>>>> "timestamp": "2015-06-08T15:37:24.449217401-0400" }
>>>>> $.:
>>>>> $/:
>>>>> --------
>>>>> This makes sense - the mmjsonparse populates the $! element.
>>>>>
>>>>> (2) The second mode was with the mmexternal action at # 2 enabled and the
>>>>> DebugFormat action at # 3 - this results in rsyslogd suffering a segfault.
>>>>>
>>>>>
>>>>> (3) The third mode has the mmexternal action at # 2 enabled and the
>>>>> DebugFormat action at #1 enabled:
>>>>>
>>>>> Debug line with all properties:
>>>>> FROMHOST: 'dboles-victim-0', fromhost-ip: '127.0.0.1', HOSTNAME:
>>>>> 'dboles-victim-0', PRI: 12,
>>>>> syslogtag ':', programname: '', APP-NAME: '', PROCID: '-', MSGID: '-',
>>>>> TIMESTAMP: 'Jun 8 15:41:26', STRUCTURED-DATA: '-',
>>>>> msg: ' @cee:{"msg":"[umberlog test] testing ... 1, 2,
>>>>> 3\n","count":"2","el.0.type":"ladybug","el.0.name":"natalie","el.0.spots":"12","el.1.type":"ant","el.1.name":"B289","el.1.role":"worker","el.1.age":"9","pid":"1828","facility":"user","priority":"warn","uid":"1002","gid":"1002","host":"dboles-victim-0","program":"","timestamp":"2015-06-08T15:41:26.302102005-0400"}'
>>>>> escaped msg: ' @cee:{"msg":"[umberlog test] testing ... 1, 2,
>>>>> 3\n","count":"2","el.0.type":"ladybug","el.0.name":"natalie","el.0.spots":"12","el.1.type":"ant","el.1.name":"B289","el.1.role":"worker","el.1.age":"9","pid":"1828","facility":"user","priority":"warn","uid":"1002","gid":"1002","host":"dboles-victim-0","program":"","timestamp":"2015-06-08T15:41:26.302102005-0400"}'
>>>>> inputname: imuxsock rawmsg: '<12>Jun 8 15:41:26 : @cee:{"msg":"[umberlog
>>>>> test] testing ... 1, 2,
>>>>> 3\n","count":"2","el.0.type":"ladybug","el.0.name":"natalie","el.0.spots":"12","el.1.type":"ant","el.1.name":"B289","el.1.role":"worker","el.1.age":"9","pid":"1828","facility":"user","priority":"warn","uid":"1002","gid":"1002","host":"dboles-victim-0","program":"","timestamp":"2015-06-08T15:41:26.302102005-0400"}'
>>>>> $!:
>>>>> $.:
>>>>> $/:
>>>>>
>>>>> --------
>>>>>
>>>>> You also asked for what my external Python script gets as input:
>>>>>
>>>>> { "msg": " @cee:{\"msg\":\"[umberlog test] testing ... 1, 2,
>>>>> 3\\n\",\"count\":\"2\",\"el.0.type\":\"
>>>>> ladybug\",\"el.0.name\":\"natalie\",\"el.0.spots\":\"12\",\"el.1.type\":\"ant\",\"el.1.name\":\"B289
>>>>> \",\"el.1.role\":\"worker\",\"el.1.age\":\"9\",\"pid\":\"4440\",\"facility\":\"user\",\"priority\":\
>>>>> "warn\",\"uid\":\"1002\",\"gid\":\"1002\",\"host\":\"dboles-victim-0\",\"program\":\"\",\"timestamp\
>>>>> ":\"2015-06-08T16:28:12.877502675-0400\"}", "rawmsg": "<12>Jun 8
>>>>> 16:28:12 : @cee:{\"msg\":\"[umberl
>>>>> og test] testing ... 1, 2,
>>>>> 3\\n\",\"count\":\"2\",\"el.0.type\":\"ladybug\",\"el.0.name\":\"natalie\
>>>>> ",\"el.0.spots\":\"12\",\"el.1.type\":\"ant\",\"el.1.name\":\"B289\",\"el.1.role\":\"worker\",\"el.1
>>>>> .age\":\"9\",\"pid\":\"4440\",\"facility\":\"user\",\"priority\":\"warn\",\"uid\":\"1002\",\"gid\":\
>>>>> "1002\",\"host\":\"dboles-victim-0\",\"program\":\"\",\"timestamp\":\"2015-06-08T16:28:12.877502675-
>>>>> 0400\"}", "timereported": "2015-06-08T16:28:12.877616-04:00", "hostname":
>>>>> "dboles-victim-0", "syslog
>>>>> tag": ":", "inputname": "imuxsock", "fromhost": "dboles-victim-0",
>>>>> "fromhost-ip": "127.0.0.1", "pri"
>>>>> : "12", "syslogfacility": "1", "syslogseverity": "4", "timegenerated":
>>>>> "2015-06-08T16:28:12.877616-0
>>>>> 4:00", "programname": "", "protocol-version": "0", "structured-data":
>>>>> "-", "app-name": "", "procid":
>>>>> "-", "msgid": "-", "uuid": null, "$!": { "msg": "[umberlog test] testing
>>>>> ... 1, 2, 3\n", "count": "
>>>>> 2", "el.0.type": "ladybug", "el.0.name": "natalie", "el.0.spots": "12",
>>>>> "el.1.type": "ant", "el.1.na
>>>>> me": "B289", "el.1.role": "worker", "el.1.age": "9", "pid": "4440",
>>>>> "facility": "user", "priority":
>>>>> "warn", "uid": "1002", "gid": "1002", "host": "dboles-victim-0",
>>>>> "program": "", "timestamp": "2015-0
>>>>> 6-08T16:28:12.877502675-0400" } }
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ________________________________________
>>>>> From: [email protected]
>>>>> [[email protected]] on behalf of David Lang
>>>>> [[email protected]]
>>>>> Sent: Monday, June 01, 2015 5:06 PM
>>>>> To: rsyslog-users
>>>>> Subject: Re: [rsyslog] Unable to use foreach
>>>>>
>>>>> On Sun, 24 May 2015, David Boles (dboles) wrote:
>>>>>
>>>>>> Thanks Rainer / David,
>>>>>>
>>>>>> Based on Rainer's input I've pivoted to going down the mmexternal path
>>>>>> and have questions based on experimenting with that.
>>>>>>
>>>>>> To answer David's "what are you trying to do" question and make things
>>>>>> as simple and concrete as possible I've created a clean example (log
>>>>>> entry generator, rsyslog configuration, mmexternal plugin, etc.) that
>>>>>> captures what I'm trying to do. To avoid pasting a bunch of stuff into
>>>>>> these emails I set up a git repo with the example at:
>>>>>>
>>>>>> https://github.com/davidboles/snowflake.git
>>>>>>
>>>>>> The README.md explains the elements of that repo. I have the following
>>>>>> questions:
>>>>>>
>>>>>> (1) The generated result does not have the { "hostname" : "frodo" }
>>>>>> anywhere
>>>>>> in it. The docs say that I can add elements to the JSON elements of the
>>>>>> message, although that sentence is ambiguous as to whether I can do so
>>>>>> if I
>>>>>> haven't modified some other element.
>>>>>
>>>>> sorry for the delay in responding
>>>>>
>>>>> in rsyslog, you don't have many elements. There are only a handful of
>>>>> built-in
>>>>> properties and then everything else is in $!
>>>>>
>>>>> so if you want to modify anything in $!, you must output a completely new
>>>>> $!
>>>>> string (which is the JSON representation of the tree.
>>>>>
>>>>> If you are wanting to add a new item "elements", that means that you are
>>>>> going
>>>>> to modify the $! variable to include your new element.
>>>>>
>>>>> the section talking about "if you modify the message variable tree" is
>>>>> saying
>>>>> that if you don't change $!, you can't add new variables, because all
>>>>> variables
>>>>> are inside of $!.
>>>>>
>>>>> although, since this talks about the inability to delete things, it may
>>>>> be that
>>>>> what you output is merged with what's already in $!. I'd have to test
>>>>> this (see
>>>>> how to test below)
>>>>>
>>>>>> (2) If I uncomment the template line (line 20 of build.conf) then syslog
>>>>>> SEGV's. How do I access the jsonmesg content after getting stuff back
>>>>>> through
>>>>>> mmexternal?
>>>>>
>>>>> it's not line 20 in the currently visible file, but the best way to
>>>>> troubleshoot
>>>>> "how do I access blah" is to write a file with the format
>>>>> RSYSLOG_DebugFormat
>>>>> and it will show you all the variables that are defined at that point. In
>>>>> a
>>>>> situation like yours where you are doing multiple parse actions, it will
>>>>> show
>>>>> you the variables as of that point in the processing.
>>>>>
>>>>> so, to see how to access the variables that were set with mmexternal,
>>>>> after that
>>>>> action add another one
>>>>> /var/log/test-debug;RSYSLOG_DebugFormat
>>>>>
>>>>> and then look at the $!: line in that debug output. It will show you
>>>>> exactly
>>>>> what variables are defined at that point.
>>>>>
>>>>> I suspect that what you are outputting doesn't quite match
>>>>>
>>>>>> (3) The field "MUTABLE" has the value "Alpha" in the input to the
>>>>>> mmexternal
>>>>>> unit (I can see it in /tmp/rewriter.trace entries). What JSON should my
>>>>>> rewriter.py emit to cause that to be rewritten to "Beta"?
>>>>>
>>>>> I think it would be useful for you to show what the string is that
>>>>> rewriter.py
>>>>> receives, what it outputs, and what the resulting debug is. I think that
>>>>> will
>>>>> make it pretty obvious as to what's happening
>>>>>
>>>>> David Lang
>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> David Boles
>>>>>>
>>>>>>
>>>>>>
>>>>>> ________________________________________
>>>>>> From: [email protected]
>>>>>> [[email protected]] on behalf of Rainer Gerhards
>>>>>> [[email protected]]
>>>>>> Sent: Friday, May 22, 2015 5:05 PM
>>>>>> To: rsyslog-users
>>>>>> Subject: Re: [rsyslog] Unable to use foreach
>>>>>>
>>>>>> External plugin interface:
>>>>>> https://github.com/rsyslog/rsyslog/blob/master/plugins/external/INTERFACE.md
>>>>>>
>>>>>> Overview :
>>>>>> https://github.com/rsyslog/rsyslog/blob/master/plugins/external/README.md
>>>>>>
>>>>>> Sent from phone, thus brief.
>>>>>> _______________________________________________
>>>>>> rsyslog mailing list
>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>> http://www.rsyslog.com/professional-services/
>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>>>>> DON'T LIKE THAT.
>>>>>>
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com/professional-services/
>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>>>> DON'T LIKE THAT.
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com/professional-services/
>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>>>> DON'T LIKE THAT.
>>>>>
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>>> DON'T LIKE THAT.
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>>> DON'T LIKE THAT.
>>>
>>>
>>>
>>> --
>>> Regards,
>>> Janmejay
>>> http://codehunk.wordpress.com
>>
>>
>>
>> --
>> Regards,
>> Janmejay
>> http://codehunk.wordpress.com
>
>
>
> --
> Regards,
> Janmejay
> http://codehunk.wordpress.com
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
