It actually wasn't an omelasticsearch issue, the problem was related
to actions and call statements not being initialized correctly when
inside foreach body.
It'd have failed with omfile as well, when used as an async-action.
BTW, omelasticsearch has more file-grained error-reporting controls
now. It allows one to configure it it should print only erroneous
messages and allows interleaving of request and response payloads etc.
I am not sure if it is released yet, 8.9.0 should have it for sure.
You may want to check it.
I'll post a PR to rsyslog/master once I finish the documentation
enhancement, until next release you probably would want to merge that
PR on top of 8.8.0 (or 8.9.0) tag.
I'll update this thread once PR is ready
On Tue, Apr 14, 2015 at 2:43 PM, chenlin rao <rao.chen...@gmail.com> wrote:
> Great! The problem is fixed!
>
> BTW: the `omelasticsearch` would cause segmentfault when it got error
> response and errorfile option no define and not running with valgrind.
>
> I think we might change this because errorfile record all bulk body
> including the right ones. Then the file size grow so quickly. I may want
> off the errorfile record and force rsyslogd running. (I already know the
> *fail* mappingexception reason but has no plan to resolve it because I
> don't manage the log source, so just don't care)
>
> 2015-04-14 13:51 GMT+08:00 singh.janmejay <singh.janme...@gmail.com>:
>
>> Chenlin, can you please try again with 'master' from my fork?
>>
>> Also, please add the following param to omelasticsearch action invocation:
>>
>> --
>> action.copyMsg="on"
>> --
>>
>> I'll create a pull request(need to do some documentation change to
>> cover code changes) once you confirm the problem is fixed.
>>
>> On Mon, Apr 13, 2015 at 10:25 PM, singh.janmejay
>> <singh.janme...@gmail.com> wrote:
>> > Have the problem identified, will post you the fix tomorrow. Couldn't
>> spend
>> > enough time on it today.
>> >
>> > --
>> > Regards,
>> > Janmejay
>> >
>> > PS: Please blame the typos in this mail on my phone's uncivilized soft
>> > keyboard sporting it's not-so-smart-assist technology.
>> >
>> >
>> > On Apr 13, 2015 7:47 PM, "singh.janmejay" <singh.janme...@gmail.com>
>> wrote:
>> >>
>> >> I was able to reproduce this. Here is the trace:
>> >>
>> >> #0 0x0000003f6f4093a0 in pthread_mutex_lock () from
>> >> /lib64/libpthread.so.0
>> >> #1 0x0000000000444ca3 in qqueueEnqMsg (pThis=0x6e9a80,
>> >> flowCtlType=eFLOWCTL_NO_DELAY, pMsg=0x7ffff0002500) at queue.c:2856
>> >> #2 0x0000000000450afe in doSubmitToActionQ (pAction=0x6e93d0,
>> >> pWti=0x6cf260, pMsg=0x7ffff0002500) at ../action.c:1451
>> >> #3 0x000000000044655c in execAct (stmt=0x6e91c0, pMsg=0x7ffff0002500,
>> >> pWti=0x6cf260) at ruleset.c:197
>> >> #4 0x0000000000446f4c in scriptExec (root=0x6e91c0,
>> >> pMsg=0x7ffff0002500, pWti=0x6cf260) at ruleset.c:449
>> >> #5 0x00000000004468a5 in execForeach (stmt=0x6e9230,
>> >> pMsg=0x7ffff0002500, pWti=0x6cf260) at ruleset.c:284
>> >> #6 0x0000000000447011 in scriptExec (root=0x6e9230,
>> >> pMsg=0x7ffff0002500, pWti=0x6cf260) at ruleset.c:464
>> >> #7 0x000000000044676f in execIf (stmt=0x6e9dc0, pMsg=0x7ffff0002500,
>> >> pWti=0x6cf260) at ruleset.c:257
>> >> #8 0x0000000000446fe8 in scriptExec (root=0x6e6240,
>> >> pMsg=0x7ffff0002500, pWti=0x6cf260) at ruleset.c:461
>> >> #9 0x000000000044665a in execCall (stmt=0x6e9070,
>> >> pMsg=0x7ffff0002500, pWti=0x6cf260) at ruleset.c:232
>> >> #10 0x0000000000446fbf in scriptExec (root=0x6e9070,
>> >> pMsg=0x7ffff0002500, pWti=0x6cf260) at ruleset.c:458
>> >> #11 0x0000000000447189 in processBatch (pBatch=0x6cf298,
>> >> pWti=0x6cf260) at ruleset.c:503
>> >> #12 0x000000000045d100 in msgConsumer (notNeeded=0x0, pBatch=0x6cf298,
>> >> pWti=0x6cf260) at rsyslogd.c:575
>> >> #13 0x000000000044216b in ConsumerReg (pThis=0x6e8cb0, pWti=0x6cf260)
>> >> at queue.c:1897
>> >> #14 0x000000000043d422 in wtiWorker (pThis=0x6cf260) at wti.c:334
>> >> #15 0x000000000043bfc6 in wtpWorker (arg=0x6cf260) at wtp.c:389
>> >> #16 0x0000003f6f4079d1 in start_thread () from /lib64/libpthread.so.0
>> >> #17 0x0000003f6f0e886d in clone () from /lib64/libc.so.6
>> >>
>> >>
>> >> Line numbers for some frames are slightly different, but it seems to
>> >> be the same failure.
>> >>
>> >> On Thu, Apr 9, 2015 at 5:10 PM, chenlin rao <rao.chen...@gmail.com>
>> wrote:
>> >> > I had already modify my config to:
>> >> >
>> >> > ```
>> >> > action ( type="mmjsonparse"
>> >> > name="action_jsonarray-parse"
>> >> > )
>> >> > if ( re_match($msg, ']}$') ) then {
>> >> > foreach ($.line in $!msg) do {
>> >> > action (...
>> >> > ```
>> >> >
>> >> > just after you told me async queue no help with mm**.
>> >> >
>> >> > So, the failure was just the one with synchronously calling
>> mmjsonparse.
>> >> >
>> >> > 2015-04-09 19:19 GMT+08:00 singh.janmejay <singh.janme...@gmail.com>:
>> >> >
>> >> >> Can you reproduce the failure with synchronously calling mmjsonparse?
>> >> >>
>> >> >> I'll try to reproduce it again.
>> >> >>
>> >> >> --
>> >> >> Regards,
>> >> >> Janmejay
>> >> >>
>> >> >> PS: Please blame the typos in this mail on my phone's uncivilized
>> soft
>> >> >> keyboard sporting it's not-so-smart-assist technology.
>> >> >>
>> >> >> On Apr 9, 2015 3:20 PM, "chenlin rao" <rao.chen...@gmail.com> wrote:
>> >> >>
>> >> >> > janmyjay:
>> >> >> > I don't think that's a json-c problem, because if I use omfile
>> >> >> inside
>> >> >> > foreach, rsyslogd never died.
>> >> >> > I also try to compile and run your repo, process fault again
>> and
>> >> >> > the
>> >> >> > valgrind output keep totally same.
>> >> >> >
>> >> >> > 2015-04-08 21:26 GMT+08:00 singh.janmejay <
>> singh.janme...@gmail.com>:
>> >> >> >
>> >> >> > > BTW, here is the gdb-session-trace if you are interested:
>> >> >> > >
>> >> >> > > Program received signal SIGSEGV, Segmentation fault.
>> >> >> > > [Switching to Thread 0x7ffff1657700 (LWP 14762)]
>> >> >> > > 0x00007ffff718c23b in array_list_get_idx () from
>> >> >> > > /lib/x86_64-linux-gnu/libjson-c.so.2
>> >> >> > > (gdb) bt
>> >> >> > > #0 0x00007ffff718c23b in array_list_get_idx () from
>> >> >> > > /lib/x86_64-linux-gnu/libjson-c.so.2
>> >> >> > > #1 0x00000000004450af in execForeach (stmt=0x6ffab0,
>> >> >> > > pMsg=0x7fffec016040, pWti=0x718810) at ruleset.c:279
>> >> >> > > #2 0x0000000000445874 in scriptExec (root=0x6ffab0,
>> >> >> > > pMsg=0x7fffec016040, pWti=0x718810) at ruleset.c:464
>> >> >> > > #3 0x0000000000444fdb in execIf (stmt=0x6ffdc0,
>> >> >> > > pMsg=0x7fffec016040,
>> >> >> > > pWti=0x718810) at ruleset.c:257
>> >> >> > > #4 0x000000000044584a in scriptExec (root=0x6fd560,
>> >> >> > > pMsg=0x7fffec016040, pWti=0x718810) at ruleset.c:461
>> >> >> > > #5 0x0000000000444ebc in execCall (stmt=0x6fd4c0,
>> >> >> > > pMsg=0x7fffec016040, pWti=0x718810) at ruleset.c:232
>> >> >> > > #6 0x0000000000445820 in scriptExec (root=0x6fd4c0,
>> >> >> > > pMsg=0x7fffec016040, pWti=0x718810) at ruleset.c:458
>> >> >> > > #7 0x00000000004459f2 in processBatch (pBatch=0x718848,
>> >> >> > > pWti=0x718810) at ruleset.c:503
>> >> >> > > #8 0x000000000045b390 in msgConsumer (notNeeded=0x0,
>> >> >> > > pBatch=0x718848,
>> >> >> > > pWti=0x718810) at rsyslogd.c:575
>> >> >> > > #9 0x00000000004409f3 in ConsumerReg (pThis=0x718380,
>> >> >> > > pWti=0x718810)
>> >> >> > > at queue.c:1897
>> >> >> > > #10 0x000000000043bcfd in wtiWorker (pThis=0x718810) at wti.c:334
>> >> >> > > #11 0x000000000043a8af in wtpWorker (arg=0x718810) at wtp.c:389
>> >> >> > > #12 0x00007ffff79ac0a4 in start_thread (arg=0x7ffff1657700) at
>> >> >> > > pthread_create.c:309
>> >> >> > > #13 0x00007ffff6abc04d in clone () at
>> >> >> > > ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
>> >> >> > > (gdb) up 1
>> >> >> > > #1 0x00000000004450af in execForeach (stmt=0x6ffab0,
>> >> >> > > pMsg=0x7fffec016040, pWti=0x718810) at ruleset.c:279
>> >> >> > > 279 curr = json_object_array_get_idx(arr, i);
>> >> >> > > (gdb) p pMsg->json
>> >> >> > > $1 = (struct json_object *) 0x7fffe00176b0
>> >> >> > > (gdb) p *pMsg->json
>> >> >> > > $2 = {o_type = json_type_object, _delete = 0x7ffff718c9a0,
>> >> >> > > _to_json_string = 0x7ffff718cee0, _ref_count = 1, _pb = 0x0, o =
>> >> >> > > {c_boolean = -536774912,
>> >> >> > > c_double = 6.9533292876306475e-310, c_int64 =
>> 140736951580416,
>> >> >> > > c_object = 0x7fffe0017700, c_array = 0x7fffe0017700, c_string =
>> >> >> > > {str =
>> >> >> > > 0x7fffe0017700 "\020", len = 0}},
>> >> >> > > _user_delete = 0x0, _userdata = 0x0}
>> >> >> > > (gdb) p *pMsg->json->o.c_object
>> >> >> > > $3 = {size = 16, count = 1, collisions = 0, resizes = 0, lookups
>> =
>> >> >> > > 2,
>> >> >> > > inserts = 1, deletes = 0, name = 0x0, head = 0x7fffe00177a0,
>> tail =
>> >> >> > > 0x7fffe00177a0, table = 0x7fffe0017760,
>> >> >> > > free_fn = 0x7ffff718cca0, hash_fn = 0x7ffff718fd60
>> >> >> > > <lh_char_hash>,
>> >> >> > > equal_fn = 0x7ffff71901f0 <lh_char_equal>}
>> >> >> > > (gdb) p *pMsg->json->o.c_object->head->key
>> >> >> > > There is no member named key.
>> >> >> > > (gdb) p pMsg->json->o.c_object->head->key
>> >> >> > > There is no member named key.
>> >> >> > > (gdb) p *pMsg->json->o.c_object->head
>> >> >> > > $4 = {k = 0x7fffe0018230, v = 0x7fffe0017970, next = 0x0, prev =
>> >> >> > > 0x0}
>> >> >> > > (gdb) p *pMsg->json->o.c_object->head->k
>> >> >> > > Attempt to dereference a generic pointer.
>> >> >> > > (gdb) p (char*) pMsg->json->o.c_object->head->k
>> >> >> > > $5 = 0x7fffe0018230 "msg"
>> >> >> > > (gdb) p (json_object*) pMsg->json->o.c_object->head->v
>> >> >> > > $6 = (json_object *) 0x7fffe0017970
>> >> >> > > (gdb) p *(json_object*) pMsg->json->o.c_object->head->v
>> >> >> > > $7 = {o_type = json_type_string, _delete = 0x7ffff718c9c0,
>> >> >> > > _to_json_string = 0x7ffff718c930, _ref_count = 1, _pb = 0x0, o =
>> >> >> > > {c_boolean = -536774208,
>> >> >> > > c_double = 6.9533292876654297e-310, c_int64 =
>> 140736951581120,
>> >> >> > > c_object = 0x7fffe00179c0, c_array = 0x7fffe00179c0, c_string = {
>> >> >> > > str = 0x7fffe00179c0 "[{\"content\":\"Error
>> >> >> > > Domain=WBBussinessErrorDomain Code=20030 \\\"the user base info
>> is
>> >> >> > > not
>> >> >> > > exists!\\\" UserInfo=0x17127cc00 {NSLocalizedDescription=the user
>> >> >> > > base
>> >> >> > > info is not exists!, WBBussinessErrorInfoKe"..., len = 2144}},
>> >> >> > > _user_delete = 0x0, _userdata = 0x0}
>> >> >> > >
>> >> >> > > On Wed, Apr 8, 2015 at 6:53 PM, singh.janmejay <
>> >> >> singh.janme...@gmail.com
>> >> >> > >
>> >> >> > > wrote:
>> >> >> > > > Spent some time on this today.
>> >> >> > > > I had the crash reproducing consistently with tcpflood -C 1000
>> >> >> > > > ...
>> >> >> > > >
>> >> >> > > > Here is what I observed:
>> >> >> > > > - When large number of messages (copy of the message you
>> shared)
>> >> >> > > > are
>> >> >> > > > sent, some messages get parsed with key 'msg' pointing to a
>> >> >> > > > json-string (its contents is the entire remaining string,
>> >> >> > > > starting
>> >> >> > > > with [{"content....
>> >> >> > > > - I have added a type-check which ensures foreach tries to
>> >> >> > > > iterate
>> >> >> > > > only when the given type is an array, if not, it prints out a
>> >> >> > > > warning
>> >> >> > > > and skips the loop.
>> >> >> > > >
>> >> >> > > > This means, it'll skip ill-parsed messages. _I haven't looked
>> >> >> > > > through
>> >> >> > > > the json-c parser code to reason code-flow that can lead to
>> this
>> >> >> > > > problem_, but because of it happening on a different thread,
>> >> >> > > > changes
>> >> >> > > > may be visible to ruleset thread in unexpected order, which
>> would
>> >> >> make
>> >> >> > > > it see the json-object being available, but its state being
>> >> >> > > > intermediate (rather than final).
>> >> >> > > >
>> >> >> > > > Can you test with git://github.com/janmejay/rsyslog.git and
>> let
>> >> >> > > > us
>> >> >> > > > know if it fixes the crash?
>> >> >> > > >
>> >> >> > > > With this change I can't reproduce the crash anymore, but
>> again,
>> >> >> > > > because modifications to the json-object are happening on a
>> >> >> > > > different
>> >> >> > > > thread, it could lead to crash on accessing that object for a
>> >> >> > > > different reason. However, this also means the ill-parsed
>> >> >> > > > messages
>> >> >> > > > will be skipped, so depending on how you reason about it, they
>> >> >> > > > will
>> >> >> be
>> >> >> > > > lost.
>> >> >> > > >
>> >> >> > > > While this fixes the value not being an array problem, you
>> still
>> >> >> > > > want
>> >> >> > > > to move to sync mmjsonparse call.
>> >> >> > > >
>> >> >> > > >
>> >> >> > > >
>> >> >> > > > On Mon, Apr 6, 2015 at 6:07 PM, chenlin rao
>> >> >> > > > <rao.chen...@gmail.com>
>> >> >> > > wrote:
>> >> >> > > >> yes.
>> >> >> > > >> rsyslogd 8.8.0.ad1, compiled with:
>> >> >> > > >> PLATFORM: x86_64-redhat-linux-gnu
>> >> >> > > >> PLATFORM (lsb_release -d):
>> >> >> > > >> FEATURE_REGEXP: Yes
>> >> >> > > >> GSSAPI Kerberos 5 support: No
>> >> >> > > >> FEATURE_DEBUG (debug build, slow code): No
>> >> >> > > >> 32bit Atomic operations supported: Yes
>> >> >> > > >> 64bit Atomic operations supported: Yes
>> >> >> > > >> memory allocator: system default
>> >> >> > > >> Runtime Instrumentation (slow code): No
>> >> >> > > >> uuid support: Yes
>> >> >> > > >> Number of Bits in RainerScript integers: 64
>> >> >> > > >>
>> >> >> > > >> 2015-04-06 20:32 GMT+08:00 singh.janmejay
>> >> >> > > >> <singh.janme...@gmail.com
>> >> >> >:
>> >> >> > > >>
>> >> >> > > >>> So it is 8.8.0?
>> >> >> > > >>>
>> >> >> > > >>> On Mon, Apr 6, 2015 at 5:33 PM, chenlin rao
>> >> >> > > >>> <rao.chen...@gmail.com
>> >> >> >
>> >> >> > > wrote:
>> >> >> > > >>> > I install latest version from rsyslog_v8 yum repo in
>> >> >> > > >>> > CentOS6.5.
>> >> >> > > >>> > And for more options in valgrind report, I compiled another
>> >> >> version
>> >> >> > > from
>> >> >> > > >>> > git master, ran into the same problem.
>> >> >> > > >>> >
>> >> >> > > >>> > 2015-04-06 13:55 GMT+08:00 singh.janmejay <
>> >> >> > singh.janme...@gmail.com
>> >> >> > > >:
>> >> >> > > >>> >
>> >> >> > > >>> >> It was looking very similar to the issue that Thomas
>> >> >> > > >>> >> reported on
>> >> >> > > >>> >>
>> >> >> > > >>> >>
>> >> >> > > >>>
>> >> >> > >
>> >> >> >
>> >> >>
>> >> >>
>> http://www.gossamer-threads.com/lists/rsyslog/users/16204?do=post_view_threaded
>> >> >> > > >>> >>
>> >> >> > > >>> >> To double-check, I cherry-picked the two patches in that
>> PR,
>> >> >> which
>> >> >> > > >>> >> fixed the failing json_array_looping test.
>> >> >> > > >>> >>
>> >> >> > > >>> >> So I'll now focus on Chenlin's problem.
>> >> >> > > >>> >>
>> >> >> > > >>> >> To rule-out the same problem (the one that PR fixes),
>> >> >> > > >>> >> Chenlin,
>> >> >> can
>> >> >> > > you
>> >> >> > > >>> >> confirm which version of rsyslog are you on?
>> >> >> > > >>> >>
>> >> >> > > >>> >>
>> >> >> > > >>> >>
>> >> >> > > >>> >> On Thu, Apr 2, 2015 at 6:58 PM, singh.janmejay <
>> >> >> > > >>> singh.janme...@gmail.com>
>> >> >> > > >>> >> wrote:
>> >> >> > > >>> >> > I couldn't spend too much time on it, but just saw it
>> >> >> reproduce
>> >> >> > > on my
>> >> >> > > >>> >> > local env (also running Ubuntu 12.04).
>> >> >> > > >>> >> >
>> >> >> > > >>> >> > Will again get a chance to look at in a a few days
>> (likely
>> >> >> > > Monday).
>> >> >> > > >>> >> >
>> >> >> > > >>> >> > Again, sorry for the delay.
>> >> >> > > >>> >> >
>> >> >> > > >>> >> > On Thu, Apr 2, 2015 at 6:43 PM, singh.janmejay <
>> >> >> > > >>> singh.janme...@gmail.com>
>> >> >> > > >>> >> wrote:
>> >> >> > > >>> >> >> Foreach support is a recent addition.
>> >> >> > > >>> >> >>
>> >> >> > > >>> >> >> On Thu, Apr 2, 2015 at 3:34 PM, singh.janmejay <
>> >> >> > > >>> >> singh.janme...@gmail.com> wrote:
>> >> >> > > >>> >> >>> I saw that mail, but by that time I had already setup
>> >> >> > > >>> >> >>> the
>> >> >> > > >>> environment
>> >> >> > > >>> >> >>> to reproduce it with Chenlin's config.
>> >> >> > > >>> >> >>>
>> >> >> > > >>> >> >>> I should pursue the CI failure first. Didn't know it
>> was
>> >> >> > > happening
>> >> >> > > >>> in
>> >> >> > > >>> >> every run.
>> >> >> > > >>> >> >>>
>> >> >> > > >>> >> >>> On Thu, Apr 2, 2015 at 2:04 PM, Rainer Gerhards
>> >> >> > > >>> >> >>> <rgerha...@hq.adiscon.com> wrote:
>> >> >> > > >>> >> >>>> I
>> >> >> > > >>> >> >>>>
>> >> >> > > >>> >> >>>> 2015-04-02 7:35 GMT+02:00 singh.janmejay <
>> >> >> > > singh.janme...@gmail.com
>> >> >> > > >>> >:
>> >> >> > > >>> >> >>>>> Hadn't noticed the use of msg key early-on, so "]}"
>> >> >> > shouldn't
>> >> >> > > be a
>> >> >> > > >>> >> >>>>> problem, but I can't reproduce it with input i
>> crafted
>> >> >> > > >>> >> >>>>> to
>> >> >> > work
>> >> >> > > >>> with
>> >> >> > > >>> >> >>>>> it.
>> >> >> > > >>> >> >>>>> Here is the crafted input:
>> >> >> > > >>> >> >>>>> @cee:{"msg" : ["G\"foo\":\"10\"}",
>> "G\"bar\":\"20\""]}
>> >> >> > > >>> >> >>>>>
>> >> >> > > >>> >> >>>>> This is what does fail it, im using a bad type (it
>> >> >> > > >>> >> >>>>> doesn't
>> >> >> > do
>> >> >> > > >>> >> type-check yet):
>> >> >> > > >>> >> >>>>> @cee:{"msg" : "G\"foo\":\"10\"}", "bar" : ["foo"]}
>> >> >> > > >>> >> >>>>>
>> >> >> > > >>> >> >>>>> Here it ending in value of 'bar' satisfies the ']}'
>> >> >> > > conditional,
>> >> >> > > >>> but
>> >> >> > > >>> >> >>>>> msg isn't an array. I'll add array-typecheck, but
>> >> >> > > >>> >> >>>>> other
>> >> >> than
>> >> >> > > this
>> >> >> > > >>> I
>> >> >> > > >>> >> >>>>> have been unable to reproduce it, so this may not
>> fix
>> >> >> > > >>> >> >>>>> your
>> >> >> > > >>> problem.
>> >> >> > > >>> >> >>>>>
>> >> >> > > >>> >> >>>>> I think your input line (or entire input file) will
>> >> >> > > >>> >> >>>>> help.
>> >> >> > > >>> >> >>>>
>> >> >> > > >>> >> >>>> No sure if you overlooked that posting. As it looks,
>> >> >> > > >>> >> >>>> the
>> >> >> > > problem
>> >> >> > > >>> seems
>> >> >> > > >>> >> >>>> to be reproducible with the testbench as well. At
>> least
>> >> >> this
>> >> >> > > commit
>> >> >> > > >>> >> >>>> continously fails:
>> >> >> > > >>> >> >>>>
>> >> >> > > >>> >> >>>> https://github.com/rsyslog/rsyslog/pull/286
>> >> >> > > >>> >> >>>>
>> >> >> > > >>> >> >>>> I've also sent a build log via te ML a couple of days
>> >> >> > > >>> >> >>>> ago.
>> >> >> > > >>> >> >>>>
>> >> >> > > >>> >> >>>> HTH
>> >> >> > > >>> >> >>>> Rainer
>> >> >> > > >>> >> >>>>>
>> >> >> > > >>> >> >>>>> On Thu, Apr 2, 2015 at 9:39 AM, singh.janmejay <
>> >> >> > > >>> >> singh.janme...@gmail.com> wrote:
>> >> >> > > >>> >> >>>>>> Hi Chenlin,
>> >> >> > > >>> >> >>>>>>
>> >> >> > > >>> >> >>>>>> I finally have some time to look at it. (Sorry for
>> >> >> > > >>> >> >>>>>> the
>> >> >> > > delay.)
>> >> >> > > >>> >> >>>>>>
>> >> >> > > >>> >> >>>>>> Can you please share a sample log-line? Feel free
>> to
>> >> >> > > anonymize
>> >> >> > > >>> it,
>> >> >> > > >>> >> but
>> >> >> > > >>> >> >>>>>> please keep the structure and types as is.
>> >> >> > > >>> >> >>>>>>
>> >> >> > > >>> >> >>>>>> I see that you are checking $msg ends with "]}",
>> that
>> >> >> means
>> >> >> > > it
>> >> >> > > >>> is an
>> >> >> > > >>> >> >>>>>> object, not an array.
>> >> >> > > >>> >> >>>>>>
>> >> >> > > >>> >> >>>>>> If my interpretation is correct, trying to iterate
>> >> >> > > >>> >> >>>>>> over
>> >> >> > > object
>> >> >> > > >>> >> instead
>> >> >> > > >>> >> >>>>>> of array may be causing this problem. (I haven't
>> >> >> > > >>> >> >>>>>> looked
>> >> >> at
>> >> >> > > code
>> >> >> > > >>> yet
>> >> >> > > >>> >> to
>> >> >> > > >>> >> >>>>>> verify this manifestation though).
>> >> >> > > >>> >> >>>>>>
>> >> >> > > >>> >> >>>>>> On Tue, Mar 31, 2015 at 2:32 PM, Rainer Gerhards
>> >> >> > > >>> >> >>>>>> <rgerha...@hq.adiscon.com> wrote:
>> >> >> > > >>> >> >>>>>>> 2015-03-30 19:55 GMT+02:00 singh.janmejay <
>> >> >> > > >>> >> singh.janme...@gmail.com>:
>> >> >> > > >>> >> >>>>>>>> Sorry, I haven't had a chance to look at it yet.
>> >> >> > > >>> >> >>>>>>>
>> >> >> > > >>> >> >>>>>>> Take your time, I know how challenging it is at
>> >> >> > > >>> >> >>>>>>> times
>> >> >> ;) I
>> >> >> > > just
>> >> >> > > >>> >> wanted
>> >> >> > > >>> >> >>>>>>> to spread the good news that we have a good repro.
>> >> >> > > >>> >> >>>>>>>
>> >> >> > > >>> >> >>>>>>> Rainer
>> >> >> > > >>> >> >>>>>>>>
>> >> >> > > >>> >> >>>>>>>> Will get to this asap.
>> >> >> > > >>> >> >>>>>>>>
>> >> >> > > >>> >> >>>>>>>> On Mon, Mar 30, 2015 at 2:36 PM, Rainer Gerhards
>> >> >> > > >>> >> >>>>>>>> <rgerha...@hq.adiscon.com> wrote:
>> >> >> > > >>> >> >>>>>>>>> Co-incidentally, an unrelated testbench run also
>> >> >> > > experienced
>> >> >> > > >>> >> problems
>> >> >> > > >>> >> >>>>>>>>> in this area. I have attached the testbench log.
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> Here is the valgrind report:
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> 5812.499032607:main Q:Reg/w0 : wti 0x614e9e0:
>> we
>> >> >> > > >>> >> >>>>>>>>> need
>> >> >> > to
>> >> >> > > >>> create
>> >> >> > > >>> >> a new
>> >> >> > > >>> >> >>>>>>>>> action worker instance for action 2
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> 5812.499181273:main Q:Reg/w0 : Action 2
>> >> >> > > >>> >> >>>>>>>>> transitioned
>> >> >> to
>> >> >> > > >>> state:
>> >> >> > > >>> >> itx
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> 5812.499314576:main Q:Reg/w0 : FOREACH
>> .corge
>> >> >> > > >>> >> >>>>>>>>> IN
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> 5812.499625664:main Q:Reg/w0 : var
>> >> >> > > >>> >> >>>>>>>>> '.quux!bar'
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> 5812.500078770:main Q:Reg/w0 : eval expr
>> >> >> > > >>> >> >>>>>>>>> 0x60849e0,
>> >> >> > type
>> >> >> > > >>> 'V[86]'
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== Thread 6:
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== Invalid read of size 8
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== at 0x587B134: lh_table_lookup_entry
>> >> >> > > >>> >> >>>>>>>>> (in
>> >> >> > > >>> >> >>>>>>>>> /usr/lib/x86_64-linux-gnu/libjson.so.0.0.1)
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== by 0x587B1A8: lh_table_lookup (in
>> >> >> > > >>> >> >>>>>>>>> /usr/lib/x86_64-linux-gnu/libjson.so.0.0.1)
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== by 0x416F98: jsonVarExtract
>> >> >> > > >>> >> >>>>>>>>> (msg.c:4218)
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== by 0x41C82F: msgGetJSONPropJSON
>> >> >> (msg.c:2848)
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== by 0x4562D0: cnfexprEval
>> >> >> > (rainerscript.c:1785)
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== by 0x45842E: cnfexprEvalCollection
>> >> >> > > >>> >> (rainerscript.c:2483)
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== by 0x435CFC: scriptExec
>> >> >> > > >>> >> >>>>>>>>> (ruleset.c:271)
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== by 0x435D6B: scriptExec
>> >> >> > > >>> >> >>>>>>>>> (ruleset.c:284)
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== by 0x436233: processBatch
>> >> >> > > >>> >> >>>>>>>>> (ruleset.c:503)
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== by 0x44464B: msgConsumer
>> >> >> > > >>> >> >>>>>>>>> (rsyslogd.c:576)
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== by 0x432AC2: ConsumerReg
>> >> >> > > >>> >> >>>>>>>>> (queue.c:1897)
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== by 0x42F55E: wtiWorker (wti.c:334)
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== Address 0x61645c8 is 8 bytes before a
>> >> >> > > >>> >> >>>>>>>>> block
>> >> >> of
>> >> >> > > size
>> >> >> > > >>> 5
>> >> >> > > >>> >> alloc'd
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== at 0x4C2B6CD: malloc (in
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>
>> /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== by 0x5D0B081: strdup (strdup.c:43)
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== by 0x5878A5C: json_object_object_add
>> >> >> > > >>> >> >>>>>>>>> (in
>> >> >> > > >>> >> >>>>>>>>> /usr/lib/x86_64-linux-gnu/libjson.so.0.0.1)
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== by 0x41E419: msgAddJSON (msg.c:4379)
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== by 0x41F2E0: msgSetJSONFromVar
>> >> >> (msg.c:4554)
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== by 0x435D4E: scriptExec
>> >> >> > > >>> >> >>>>>>>>> (ruleset.c:283)
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== by 0x436233: processBatch
>> >> >> > > >>> >> >>>>>>>>> (ruleset.c:503)
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== by 0x44464B: msgConsumer
>> >> >> > > >>> >> >>>>>>>>> (rsyslogd.c:576)
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== by 0x432AC2: ConsumerReg
>> >> >> > > >>> >> >>>>>>>>> (queue.c:1897)
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== by 0x42F55E: wtiWorker (wti.c:334)
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== by 0x42E6C6: wtpWorker (wtp.c:389)
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== by 0x5050E99: start_thread
>> >> >> > > (pthread_create.c:308)
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284==
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== Jump to the invalid address stated on
>> the
>> >> >> next
>> >> >> > > line
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== at 0x0: ???
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== by 0x587B136: lh_table_lookup_entry
>> >> >> > > >>> >> >>>>>>>>> (in
>> >> >> > > >>> >> >>>>>>>>> /usr/lib/x86_64-linux-gnu/libjson.so.0.0.1)
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== by 0x587B1A8: lh_table_lookup (in
>> >> >> > > >>> >> >>>>>>>>> /usr/lib/x86_64-linux-gnu/libjson.so.0.0.1)
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== by 0x416F98: jsonVarExtract
>> >> >> > > >>> >> >>>>>>>>> (msg.c:4218)
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== by 0x41C82F: msgGetJSONPropJSON
>> >> >> (msg.c:2848)
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== by 0x4562D0: cnfexprEval
>> >> >> > (rainerscript.c:1785)
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== by 0x45842E: cnfexprEvalCollection
>> >> >> > > >>> >> (rainerscript.c:2483)
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== by 0x435CFC: scriptExec
>> >> >> > > >>> >> >>>>>>>>> (ruleset.c:271)
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== by 0x435D6B: scriptExec
>> >> >> > > >>> >> >>>>>>>>> (ruleset.c:284)
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== by 0x436233: processBatch
>> >> >> > > >>> >> >>>>>>>>> (ruleset.c:503)
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== by 0x44464B: msgConsumer
>> >> >> > > >>> >> >>>>>>>>> (rsyslogd.c:576)
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== by 0x432AC2: ConsumerReg
>> >> >> > > >>> >> >>>>>>>>> (queue.c:1897)
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== Address 0x0 is not stack'd, malloc'd
>> or
>> >> >> > > (recently)
>> >> >> > > >>> >> free'd
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284==
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284==
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== Process terminating with default action
>> >> >> > > >>> >> >>>>>>>>> of
>> >> >> > > signal 11
>> >> >> > > >>> >> (SIGSEGV)
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== Bad permissions for mapped region at
>> >> >> > > >>> >> >>>>>>>>> address
>> >> >> > 0x0
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== at 0x0: ???
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== by 0x587B136: lh_table_lookup_entry
>> >> >> > > >>> >> >>>>>>>>> (in
>> >> >> > > >>> >> >>>>>>>>> /usr/lib/x86_64-linux-gnu/libjson.so.0.0.1)
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== by 0x587B1A8: lh_table_lookup (in
>> >> >> > > >>> >> >>>>>>>>> /usr/lib/x86_64-linux-gnu/libjson.so.0.0.1)
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== by 0x416F98: jsonVarExtract
>> >> >> > > >>> >> >>>>>>>>> (msg.c:4218)
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== by 0x41C82F: msgGetJSONPropJSON
>> >> >> (msg.c:2848)
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== by 0x4562D0: cnfexprEval
>> >> >> > (rainerscript.c:1785)
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== by 0x45842E: cnfexprEvalCollection
>> >> >> > > >>> >> (rainerscript.c:2483)
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== by 0x435CFC: scriptExec
>> >> >> > > >>> >> >>>>>>>>> (ruleset.c:271)
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== by 0x435D6B: scriptExec
>> >> >> > > >>> >> >>>>>>>>> (ruleset.c:284)
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== by 0x436233: processBatch
>> >> >> > > >>> >> >>>>>>>>> (ruleset.c:503)
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== by 0x44464B: msgConsumer
>> >> >> > > >>> >> >>>>>>>>> (rsyslogd.c:576)
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== by 0x432AC2: ConsumerReg
>> >> >> > > >>> >> >>>>>>>>> (queue.c:1897)
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284==
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== HEAP SUMMARY:
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== in use at exit: 961,978 bytes in
>> >> >> > > >>> >> >>>>>>>>> 1,632
>> >> >> > blocks
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== total heap usage: 2,448 allocs, 816
>> >> >> > > >>> >> >>>>>>>>> frees,
>> >> >> > > >>> 1,023,184
>> >> >> > > >>> >> bytes allocated
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284==
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== LEAK SUMMARY:
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== definitely lost: 6 bytes in 1 blocks
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== indirectly lost: 0 bytes in 0 blocks
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== possibly lost: 2,592 bytes in 9
>> >> >> > > >>> >> >>>>>>>>> blocks
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== still reachable: 959,380 bytes in
>> >> >> > > >>> >> >>>>>>>>> 1,622
>> >> >> > blocks
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== suppressed: 0 bytes in 0 blocks
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== Rerun with --leak-check=full to see
>> >> >> > > >>> >> >>>>>>>>> details
>> >> >> of
>> >> >> > > leaked
>> >> >> > > >>> >> memory
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284==
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== For counts of detected and suppressed
>> >> >> > > >>> >> >>>>>>>>> errors,
>> >> >> > > rerun
>> >> >> > > >>> >> with: -v
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ==2284== ERROR SUMMARY: 2 errors from 2 contexts
>> >> >> > > (suppressed:
>> >> >> > > >>> 2
>> >> >> > > >>> >> from 2)
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> receiving response: Connection reset by peer
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> ./diag.sh: line 17: 2284 Killed
>> >> >> > > $valgrind
>> >> >> > > >>> >> >>>>>>>>> ../tools/rsyslogd -C -n -irsyslog$3.pid
>> >> >> > > >>> >> -M../runtime/.libs:../.libs
>> >> >> > > >>> >> >>>>>>>>> -f$srcdir/testsuites/$2
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> FAIL: json_array_looping.sh
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> Rainer
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> 2015-03-21 17:50 GMT+01:00 chenlin rao <
>> >> >> > > rao.chen...@gmail.com
>> >> >> > > >>> >:
>> >> >> > > >>> >> >>>>>>>>>> thanks for the information. But rsyslogd also
>> >> >> > > >>> >> >>>>>>>>>> fault
>> >> >> > > after I
>> >> >> > > >>> >> change
>> >> >> > > >>> >> >>>>>>>>>> mmjsonparse action config as `action (
>> >> >> > type="mmjsonparse"
>> >> >> > > >>> >> >>>>>>>>>> name="action_jsonarray-parse"`. output as
>> >> >> > > >>> >> >>>>>>>>>> follow:
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> 6008.997308131:main Q:Reg/w1 : eval expr
>> >> >> > > >>> >> >>>>>>>>>> 0x52f18f0,
>> >> >> > > return
>> >> >> > > >>> >> datatype 'J'
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> 6009.002050327:main Q:Reg/w1 : ACTION 16
>> >> >> > > >>> >> >>>>>>>>>> [builtin:omfwd:action(type="builtin:omfwd"
>> ...)]
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> 6009.002357554:main Q:Reg/w1 : executing
>> action
>> >> >> > > >>> >> >>>>>>>>>> 16
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> 6009.002454575:main Q:Reg/w1 : Called action,
>> >> >> logging
>> >> >> > to
>> >> >> > > >>> >> builtin:omfwd
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> ==9947== Thread 22:
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> ==9947== Invalid read of size 4
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> ==9947== at 0x3A044093A0: pthread_mutex_lock
>> >> >> > > >>> >> >>>>>>>>>> (in
>> >> >> > > /lib64/
>> >> >> > > >>> >> >>>>>>>>>> libpthread-2.12.so)
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> ==9947== by 0x438053: qqueueEnqMsg
>> >> >> > > >>> >> >>>>>>>>>> (queue.c:2856)
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> ==9947== by 0x4421AB: doSubmitToActionQ
>> >> >> > > (action.c:1403)
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> ==9947== by 0x439D6C: scriptExec
>> >> >> > > >>> >> >>>>>>>>>> (ruleset.c:197)
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> ==9947== by 0x439ECC: scriptExec
>> >> >> > > >>> >> >>>>>>>>>> (ruleset.c:284)
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> ==9947== by 0x439CD9: scriptExec
>> >> >> > > >>> >> >>>>>>>>>> (ruleset.c:416)
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> ==9947== by 0x439CD9: scriptExec
>> >> >> > > >>> >> >>>>>>>>>> (ruleset.c:416)
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> ==9947== by 0x439CD9: scriptExec
>> >> >> > > >>> >> >>>>>>>>>> (ruleset.c:416)
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> ==9947== by 0x43A40B: processBatch
>> >> >> > > >>> >> >>>>>>>>>> (ruleset.c:503)
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> ==9947== by 0x44A403: msgConsumer
>> >> >> > > >>> >> >>>>>>>>>> (rsyslogd.c:575)
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> ==9947== by 0x438D12: ConsumerReg
>> >> >> > > >>> >> >>>>>>>>>> (queue.c:1897)
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> ==9947== by 0x43314B: wtiWorker (wti.c:334)
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> ==9947== Address 0x10 is not stack'd, malloc'd
>> >> >> > > >>> >> >>>>>>>>>> or
>> >> >> > > (recently)
>> >> >> > > >>> >> free'd
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> ==9947==
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> ==9947==
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> ==9947== Process terminating with default
>> action
>> >> >> > > >>> >> >>>>>>>>>> of
>> >> >> > > signal 11
>> >> >> > > >>> >> (SIGSEGV)
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> ==9947== Access not within mapped region at
>> >> >> > > >>> >> >>>>>>>>>> address
>> >> >> > 0x10
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> ==9947== at 0x3A044093A0: pthread_mutex_lock
>> >> >> > > >>> >> >>>>>>>>>> (in
>> >> >> > > /lib64/
>> >> >> > > >>> >> >>>>>>>>>> libpthread-2.12.so)
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> ==9947== by 0x438053: qqueueEnqMsg
>> >> >> > > >>> >> >>>>>>>>>> (queue.c:2856)
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> ==9947== by 0x4421AB: doSubmitToActionQ
>> >> >> > > (action.c:1403)
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> ==9947== by 0x439D6C: scriptExec
>> >> >> > > >>> >> >>>>>>>>>> (ruleset.c:197)
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> ==9947== by 0x439ECC: scriptExec
>> >> >> > > >>> >> >>>>>>>>>> (ruleset.c:284)
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> ==9947== by 0x439CD9: scriptExec
>> >> >> > > >>> >> >>>>>>>>>> (ruleset.c:416)
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> ==9947== by 0x439CD9: scriptExec
>> >> >> > > >>> >> >>>>>>>>>> (ruleset.c:416)
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> ==9947== by 0x439CD9: scriptExec
>> >> >> > > >>> >> >>>>>>>>>> (ruleset.c:416)
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> ==9947== by 0x43A40B: processBatch
>> >> >> > > >>> >> >>>>>>>>>> (ruleset.c:503)
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> ==9947== by 0x44A403: msgConsumer
>> >> >> > > >>> >> >>>>>>>>>> (rsyslogd.c:575)
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> ==9947== by 0x438D12: ConsumerReg
>> >> >> > > >>> >> >>>>>>>>>> (queue.c:1897)
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> ==9947== by 0x43314B: wtiWorker (wti.c:334)
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> ==9947== If you believe this happened as a
>> >> >> > > >>> >> >>>>>>>>>> result
>> >> >> of a
>> >> >> > > stack
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> ==9947== overflow in your program's main
>> thread
>> >> >> > > (unlikely
>> >> >> > > >>> but
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> ==9947== possible), you can try to increase
>> the
>> >> >> > > >>> >> >>>>>>>>>> size
>> >> >> > of
>> >> >> > > the
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> ==9947== main thread stack using the
>> >> >> --main-stacksize=
>> >> >> > > flag.
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> ==9947== The main thread stack size used in
>> this
>> >> >> > > >>> >> >>>>>>>>>> run
>> >> >> > was
>> >> >> > > >>> >> 10485760.
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> ==9947==
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> ==9947== HEAP SUMMARY:
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> ==9947== in use at exit: 1,181,215,468
>> bytes
>> >> >> > > >>> >> >>>>>>>>>> in
>> >> >> > > 282,120
>> >> >> > > >>> >> blocks
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> ==9947== total heap usage: 379,556 allocs,
>> >> >> > > >>> >> >>>>>>>>>> 97,436
>> >> >> > > frees,
>> >> >> > > >>> >> 2,096,403,341
>> >> >> > > >>> >> >>>>>>>>>> bytes allocated
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> ==9947==
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> ==9947== LEAK SUMMARY:
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> ==9947== definitely lost: 516 bytes in 12
>> >> >> > > >>> >> >>>>>>>>>> blocks
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> ==9947== indirectly lost: 0 bytes in 0
>> blocks
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> ==9947== possibly lost: 33,561,793 bytes
>> in
>> >> >> > > >>> >> >>>>>>>>>> 24
>> >> >> > > blocks
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> ==9947== still reachable: 1,147,653,159
>> bytes
>> >> >> > > >>> >> >>>>>>>>>> in
>> >> >> > > 282,084
>> >> >> > > >>> >> blocks
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> ==9947== suppressed: 0 bytes in 0
>> blocks
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> ==9947== Rerun with --leak-check=full to see
>> >> >> > > >>> >> >>>>>>>>>> details
>> >> >> of
>> >> >> > > >>> leaked
>> >> >> > > >>> >> memory
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> ==9947==
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> ==9947== For counts of detected and suppressed
>> >> >> errors,
>> >> >> > > rerun
>> >> >> > > >>> >> with: -v
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> ==9947== Use --track-origins=yes to see where
>> >> >> > > uninitialised
>> >> >> > > >>> >> values come from
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> ==9947== ERROR SUMMARY: 25 errors from 5
>> contexts
>> >> >> > > >>> (suppressed:
>> >> >> > > >>> >> 100 from 9)
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> Seems very similar with before.
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> BTW: is it the "workerthreads" options make
>> sense
>> >> >> > > >>> >> >>>>>>>>>> to
>> >> >> > > >>> >> mmjsonparse/mmfields
>> >> >> > > >>> >> >>>>>>>>>> etc?
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> 2015-03-22 0:07 GMT+08:00 Rainer Gerhards <
>> >> >> > > >>> >> rgerha...@hq.adiscon.com>:
>> >> >> > > >>> >> >>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>>> 2015-03-21 14:50 GMT+01:00 chenlin rao <
>> >> >> > > >>> rao.chen...@gmail.com
>> >> >> > > >>> >> >:
>> >> >> > > >>> >> >>>>>>>>>>> > $MaxMessageSize 32m
>> >> >> > > >>> >> >>>>>>>>>>> > module( load="imtcp" )
>> >> >> > > >>> >> >>>>>>>>>>> > module( load="imuxsock" )
>> >> >> > > >>> >> >>>>>>>>>>> > module( load="imklog" )
>> >> >> > > >>> >> >>>>>>>>>>> > module( load="mmfields" )
>> >> >> > > >>> >> >>>>>>>>>>> > module( load="mmjsonparse" )
>> >> >> > > >>> >> >>>>>>>>>>> > module( load="omelasticsearch" )
>> >> >> > > >>> >> >>>>>>>>>>> > $ActionFileDefaultTemplate
>> >> >> > > RSYSLOG_TraditionalFileFormat
>> >> >> > > >>> >> >>>>>>>>>>> > $RepeatedMsgReduction off
>> >> >> > > >>> >> >>>>>>>>>>> > $WorkDirectory /data0/rsyslog
>> >> >> > > >>> >> >>>>>>>>>>> > $MainMsgQueueFilename mainQ
>> >> >> > > >>> >> >>>>>>>>>>> > $MainMsgQueueType LinkedList
>> >> >> > > >>> >> >>>>>>>>>>> > $MainMsgQueueMaxFileSize 512M
>> >> >> > > >>> >> >>>>>>>>>>> > $MainMsgQueueMaxDiskSpace 20G
>> >> >> > > >>> >> >>>>>>>>>>> > $PreserveFQDN on
>> >> >> > > >>> >> >>>>>>>>>>> > input(type="imtcp" port="514")
>> >> >> > > >>> >> >>>>>>>>>>> > template(name="local6IndexName"
>> type="list") {
>> >> >> > > >>> >> >>>>>>>>>>> > constant(value="logstash-mweibo-")
>> >> >> > > >>> >> >>>>>>>>>>> > property(name="timereported"
>> >> >> > dateFormat="rfc3339"
>> >> >> > > >>> >> position.from="1"
>> >> >> > > >>> >> >>>>>>>>>>> > position.to="4")
>> >> >> > > >>> >> >>>>>>>>>>> > constant(value=".")
>> >> >> > > >>> >> >>>>>>>>>>> > property(name="timereported"
>> >> >> > dateFormat="rfc3339"
>> >> >> > > >>> >> position.from="6"
>> >> >> > > >>> >> >>>>>>>>>>> > position.to="7")
>> >> >> > > >>> >> >>>>>>>>>>> > constant(value=".")
>> >> >> > > >>> >> >>>>>>>>>>> > property(name="timereported"
>> >> >> > dateFormat="rfc3339"
>> >> >> > > >>> >> position.from="9"
>> >> >> > > >>> >> >>>>>>>>>>> > position.to="10")
>> >> >> > > >>> >> >>>>>>>>>>> > }
>> >> >> > > >>> >> >>>>>>>>>>> > template( name="local6TypeName"
>> type="string"
>> >> >> > > >>> >> string="%programname%" )
>> >> >> > > >>> >> >>>>>>>>>>> > template( name="local6JsonArray"
>> type="list" )
>> >> >> > > >>> >> >>>>>>>>>>> > {
>> >> >> > > >>> >> >>>>>>>>>>> > constant(value="{\"@timestamp\":\"")
>> >> >> > > >>> >> property(name="timereported"
>> >> >> > > >>> >> >>>>>>>>>>> > dateFormat="rfc3339")
>> >> >> > > >>> >> >>>>>>>>>>> > constant(value="\",\"host\":\"")
>> >> >> > > >>> >> property(name="hostname")
>> >> >> > > >>> >> >>>>>>>>>>> > constant(value="\",")
>> >> >> > > >>> >> >>>>>>>>>>> > property(name="$.line"
>> >> >> > > >>> >> position.from="2")
>> >> >> > > >>> >> >>>>>>>>>>> > }
>> >> >> > > >>> >> >>>>>>>>>>> > Ruleset( name="forwardRuleSetJsonArray" )
>> >> >> > > >>> >> >>>>>>>>>>> > {
>> >> >> > > >>> >> >>>>>>>>>>> > action ( type="mmjsonparse"
>> >> >> > > >>> >> >>>>>>>>>>> > name="action_jsonarray-parse"
>> >> >> > > >>> >> >>>>>>>>>>> > queue.size="3000"
>> >> >> > > >>> >> >>>>>>>>>>> > queue.dequeuebatchsize="300"
>> >> >> > > >>> >> >>>>>>>>>>> > queue.maxdiskspace="15G"
>> >> >> > > >>> >> >>>>>>>>>>> > queue.checkpointinterval="10"
>> >> >> > > >>> >> >>>>>>>>>>> > queue.type="linkedlist"
>> >> >> > > >>> >> >>>>>>>>>>> > queue.workerthreads="30"
>> >> >> > > >>> >> >>>>>>>>>>> >
>> >> >> > > >>> >> >>>>>>>>>>> > queue.workerthreadminimummessages="100"
>> >> >> > > >>> >> >>>>>>>>>>> > queue.maxfilesize="500M"
>> >> >> > > >>> >> >>>>>>>>>>> > queue.saveonshutdown="on"
>> >> >> > > >>> >> >>>>>>>>>>> > )
>> >> >> > > >>> >> >>>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>>> running mmjsonparse on a queue
>> (asynchronously!)
>> >> >> does
>> >> >> > > not
>> >> >> > > >>> make
>> >> >> > > >>> >> any
>> >> >> > > >>> >> >>>>>>>>>>> sense. When running async, the json properties
>> >> >> > > >>> >> >>>>>>>>>>> will
>> >> >> > > never be
>> >> >> > > >>> >> seen by
>> >> >> > > >>> >> >>>>>>>>>>> the rest of the ruleset.
>> >> >> > > >>> >> >>>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>>> Maybe this even triggers the fault...
>> >> >> > > >>> >> >>>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>>> Rainer
>> >> >> > > >>> >> >>>>>>>>>>> > # maybe recieved a too large data being
>> >> >> > > >>> >> >>>>>>>>>>> > truncated
>> >> >> > > >>> >> >>>>>>>>>>> > if ( re_match($msg, ']}$') ) then {
>> >> >> > > >>> >> >>>>>>>>>>> > foreach ($.line in $!msg) do {
>> >> >> > > >>> >> >>>>>>>>>>> > action (
>> >> >> > > >>> >> >>>>>>>>>>> > # test for other output modules
>> >> >> > > >>> >> >>>>>>>>>>> > # type="omfile"
>> >> >> > > >>> file="/data/rsyslog-debug.log"
>> >> >> > > >>> >> >>>>>>>>>>> > # type="omfwd"
>> >> >> > > >>> >> >>>>>>>>>>> > Target="127.0.0.1"
>> >> >> > > >>> Port="5140"
>> >> >> > > >>> >> >>>>>>>>>>> Protocol="tcp"
>> >> >> > > >>> >> >>>>>>>>>>> > type="omelasticsearch"
>> >> >> > > >>> >> >>>>>>>>>>> > server="
>> >> >> > > >>> es.domain.com
>> >> >> > > >>> >> "
>> >> >> > > >>> >> >>>>>>>>>>> > dynSearchIndex="on"
>> >> >> > > >>> >> >>>>>>>>>>> > searchIndex="local6IndexName"
>> >> >> > > >>> >> dynSearchType="on"
>> >> >> > > >>> >> >>>>>>>>>>> > searchType="local6TypeName" asyncrepl="on"
>> >> >> > > bulkmode="on"
>> >> >> > > >>> >> >>>>>>>>>>> > template="local6JsonArray"
>> >> >> > > >>> >> >>>>>>>>>>> >
>> name="action_json2log-es1003"
>> >> >> > > >>> >> >>>>>>>>>>> >
>> >> >> > > queue.filename="action_json2log-es1003"
>> >> >> > > >>> >> >>>>>>>>>>> > queue.size="10000"
>> >> >> > > >>> >> >>>>>>>>>>> >
>> queue.dequeuebatchsize="2000"
>> >> >> > > >>> >> >>>>>>>>>>> > queue.maxdiskspace="15G"
>> >> >> > > >>> >> >>>>>>>>>>> > queue.discardseverity="3"
>> >> >> > > >>> >> >>>>>>>>>>> >
>> queue.checkpointinterval="10"
>> >> >> > > >>> >> >>>>>>>>>>> > queue.type="linkedlist"
>> >> >> > > >>> >> >>>>>>>>>>> > queue.workerthreads="5"
>> >> >> > > >>> >> >>>>>>>>>>> >
>> >> >> > > queue.workerthreadminimummessages="2000"
>> >> >> > > >>> >> >>>>>>>>>>> > queue.maxfilesize="500M"
>> >> >> > > >>> >> >>>>>>>>>>> > queue.saveonshutdown="on"
>> >> >> > > >>> >> >>>>>>>>>>> > )
>> >> >> > > >>> >> >>>>>>>>>>> > }
>> >> >> > > >>> >> >>>>>>>>>>> > stop
>> >> >> > > >>> >> >>>>>>>>>>> > }
>> >> >> > > >>> >> >>>>>>>>>>> > }
>> >> >> > > >>> >> >>>>>>>>>>> >
>> >> >> > > >>> >> >>>>>>>>>>> > if ( $programname ==
>> >> >> > 'mobile_client_net_fatal_error' )
>> >> >> > > >>> then
>> >> >> > > >>> >> >>>>>>>>>>> > {
>> >> >> > > >>> >> >>>>>>>>>>> > call forwardRuleSetJsonArray
>> >> >> > > >>> >> >>>>>>>>>>> > stop
>> >> >> > > >>> >> >>>>>>>>>>> > }
>> >> >> > > >>> >> >>>>>>>>>>> >
>> >> >> > > >>> >> >>>>>>>>>>>
>> >> >> > > >>> >>
>> >> >> > > >>>
>> >> >> > >
>> >> >> >
>> >> >>
>> >> >>
>> *.info;mail.none;authpriv.none;cron.none;local6.none;local7.none;user.none
>> >> >> > > >>> >> >>>>>>>>>>> > /var/log/messages
>> >> >> > > >>> >> >>>>>>>>>>> > authpriv.*
>> >> >> > > >>> >> /var/log/secure
>> >> >> > > >>> >> >>>>>>>>>>> > mail.*
>> >> >> > > >>> >> /var/log/maillog
>> >> >> > > >>> >> >>>>>>>>>>> > cron.*
>> >> >> > > >>> >> /var/log/cron
>> >> >> > > >>> >> >>>>>>>>>>> > uucp,news.crit
>> >> >> > > >>> >> /var/log/spooler
>> >> >> > > >>> >> >>>>>>>>>>> >
>> >> >> > > >>> >> >>>>>>>>>>> > 2015-03-21 13:14 GMT+08:00 singh.janmejay <
>> >> >> > > >>> >> singh.janme...@gmail.com>:
>> >> >> > > >>> >> >>>>>>>>>>> >
>> >> >> > > >>> >> >>>>>>>>>>> >> Can you please share your config as well?
>> >> >> > > >>> >> >>>>>>>>>>> >>
>> >> >> > > >>> >> >>>>>>>>>>> >> Also, I'll likely be able to look at it
>> only
>> >> >> after
>> >> >> > > 27th
>> >> >> > > >>> >> Mar. As of now I
>> >> >> > > >>> >> >>>>>>>>>>> >> don't have access to a computer and github
>> >> >> doesn't
>> >> >> > > seem
>> >> >> > > >>> to
>> >> >> > > >>> >> show line
>> >> >> > > >>> >> >>>>>>>>>>> >> numbers while browsing code using mobile
>> >> >> > > >>> >> >>>>>>>>>>> >> phone.
>> >> >> > > >>> >> >>>>>>>>>>> >>
>> >> >> > > >>> >> >>>>>>>>>>> >> --
>> >> >> > > >>> >> >>>>>>>>>>> >> Regards,
>> >> >> > > >>> >> >>>>>>>>>>> >> Janmejay
>> >> >> > > >>> >> >>>>>>>>>>> >>
>> >> >> > > >>> >> >>>>>>>>>>> >> PS: Please blame the typos in this mail on
>> my
>> >> >> > phone's
>> >> >> > > >>> >> uncivilized soft
>> >> >> > > >>> >> >>>>>>>>>>> >> keyboard sporting it's not-so-smart-assist
>> >> >> > > technology.
>> >> >> > > >>> >> >>>>>>>>>>> >>
>> >> >> > > >>> >> >>>>>>>>>>> >> On Mar 21, 2015 8:49 AM, "chenlin rao" <
>> >> >> > > >>> >> rao.chen...@gmail.com> wrote:
>> >> >> > > >>> >> >>>>>>>>>>> >>
>> >> >> > > >>> >> >>>>>>>>>>> >> > I try to build rsyslogd from github
>> master
>> >> >> > > >>> >> >>>>>>>>>>> >> > with
>> >> >> > > >>> >> "./configure
>> >> >> > > >>> >> >>>>>>>>>>> >> --enable-debug
>> >> >> > > >>> >> >>>>>>>>>>> >> > --enable-valgrind --enable-memcheck
>> >> >> > > >>> --enable-elasticsearch
>> >> >> > > >>> >> >>>>>>>>>>> >> > --enable-mmjsonparse --enable-mmsequence
>> >> >> > > >>> --enable-mmfields
>> >> >> > > >>> >> >>>>>>>>>>> >> > --disable-liblogging-stdlog
>> >> >> > > >>> >> >>>>>>>>>>> >> > --enable-omruleset"
>> >> >> > > >>> >> >>>>>>>>>>> >> >
>> >> >> > > >>> >> >>>>>>>>>>> >> > then process exit with:
>> >> >> > > >>> >> >>>>>>>>>>> >> >
>> >> >> > > >>> >> >>>>>>>>>>> >> > 7778.301430428:main Q[DA]:Reg/w0:
>> >> >> > > >>> >> >>>>>>>>>>> >> > ACTION 16
>> >> >> > > >>> >> >>>>>>>>>>> >> >
>> [builtin:omfwd:action(type="builtin:omfwd"
>> >> >> ...)]
>> >> >> > > >>> >> >>>>>>>>>>> >> > 7778.301676909:main Q[DA]:Reg/w0:
>> executing
>> >> >> > action
>> >> >> > > 16
>> >> >> > > >>> >> >>>>>>>>>>> >> > 7778.301791807:main Q[DA]:Reg/w0: Called
>> >> >> action,
>> >> >> > > >>> logging
>> >> >> > > >>> >> to
>> >> >> > > >>> >> >>>>>>>>>>> builtin:omfwd
>> >> >> > > >>> >> >>>>>>>>>>> >> > ==20833== Thread 2:
>> >> >> > > >>> >> >>>>>>>>>>> >> > ==20833== Invalid read of size 4
>> >> >> > > >>> >> >>>>>>>>>>> >> > ==20833== at 0x3A044093A0:
>> >> >> pthread_mutex_lock
>> >> >> > > (in
>> >> >> > > >>> >> /lib64/
>> >> >> > > >>> >> >>>>>>>>>>> >> > libpthread-2.12.so)
>> >> >> > > >>> >> >>>>>>>>>>> >> > ==20833== by 0x438053: qqueueEnqMsg
>> >> >> > > (queue.c:2856)
>> >> >> > > >>> >> >>>>>>>>>>> >> > ==20833== by 0x4421AB:
>> doSubmitToActionQ
>> >> >> > > >>> >> (action.c:1403)
>> >> >> > > >>> >> >>>>>>>>>>> >> > ==20833== by 0x439D6C: scriptExec
>> >> >> > > (ruleset.c:197)
>> >> >> > > >>> >> >>>>>>>>>>> >> > ==20833== by 0x439ECC: scriptExec
>> >> >> > > (ruleset.c:284)
>> >> >> > > >>> >> >>>>>>>>>>> >> > ==20833== by 0x439CD9: scriptExec
>> >> >> > > (ruleset.c:416)
>> >> >> > > >>> >> >>>>>>>>>>> >> > ==20833== by 0x43A40B: processBatch
>> >> >> > > (ruleset.c:503)
>> >> >> > > >>> >> >>>>>>>>>>> >> > ==20833== by 0x44A403: msgConsumer
>> >> >> > > (rsyslogd.c:575)
>> >> >> > > >>> >> >>>>>>>>>>> >> > ==20833== by 0x438D12: ConsumerReg
>> >> >> > > (queue.c:1897)
>> >> >> > > >>> >> >>>>>>>>>>> >> > ==20833== by 0x43314B: wtiWorker
>> >> >> > > >>> >> >>>>>>>>>>> >> > (wti.c:334)
>> >> >> > > >>> >> >>>>>>>>>>> >> > ==20833== by 0x431BA6: wtpWorker
>> >> >> > > >>> >> >>>>>>>>>>> >> > (wtp.c:389)
>> >> >> > > >>> >> >>>>>>>>>>> >> > ==20833== by 0x3A044079D0:
>> start_thread
>> >> >> > > >>> >> >>>>>>>>>>> >> > (in
>> >> >> > > /lib64/
>> >> >> > > >>> >> >>>>>>>>>>> libpthread-2.12.so
>> >> >> > > >>> >> >>>>>>>>>>> >> )
>> >> >> > > >>> >> >>>>>>>>>>> >> > ==20833== Address 0x10 is not stack'd,
>> >> >> malloc'd
>> >> >> > or
>> >> >> > > >>> >> (recently) free'd
>> >> >> > > >>> >> >>>>>>>>>>> >> > ==20833==
>> >> >> > > >>> >> >>>>>>>>>>> >> > ==20833==
>> >> >> > > >>> >> >>>>>>>>>>> >> > ==20833== Process terminating with
>> default
>> >> >> action
>> >> >> > > of
>> >> >> > > >>> >> signal 11
>> >> >> > > >>> >> >>>>>>>>>>> (SIGSEGV)
>> >> >> > > >>> >> >>>>>>>>>>> >> > ==20833== Access not within mapped
>> region
>> >> >> > > >>> >> >>>>>>>>>>> >> > at
>> >> >> > > address
>> >> >> > > >>> 0x10
>> >> >> > > >>> >> >>>>>>>>>>> >> > ==20833== at 0x3A044093A0:
>> >> >> pthread_mutex_lock
>> >> >> > > (in
>> >> >> > > >>> >> /lib64/
>> >> >> > > >>> >> >>>>>>>>>>> >> > libpthread-2.12.so)
>> >> >> > > >>> >> >>>>>>>>>>> >> > ==20833== by 0x438053: qqueueEnqMsg
>> >> >> > > (queue.c:2856)
>> >> >> > > >>> >> >>>>>>>>>>> >> > ==20833== by 0x4421AB:
>> doSubmitToActionQ
>> >> >> > > >>> >> (action.c:1403)
>> >> >> > > >>> >> >>>>>>>>>>> >> > ==20833== by 0x439D6C: scriptExec
>> >> >> > > (ruleset.c:197)
>> >> >> > > >>> >> >>>>>>>>>>> >> > ==20833== by 0x439ECC: scriptExec
>> >> >> > > (ruleset.c:284)
>> >> >> > > >>> >> >>>>>>>>>>> >> > ==20833== by 0x439CD9: scriptExec
>> >> >> > > (ruleset.c:416)
>> >> >> > > >>> >> >>>>>>>>>>> >> > ==20833== by 0x43A40B: processBatch
>> >> >> > > (ruleset.c:503)
>> >> >> > > >>> >> >>>>>>>>>>> >> > ==20833== by 0x44A403: msgConsumer
>> >> >> > > (rsyslogd.c:575)
>> >> >> > > >>> >> >>>>>>>>>>> >> > ==20833== by 0x438D12: ConsumerReg
>> >> >> > > (queue.c:1897)
>> >> >> > > >>> >> >>>>>>>>>>> >> > ==20833== by 0x43314B: wtiWorker
>> >> >> > > >>> >> >>>>>>>>>>> >> > (wti.c:334)
>> >> >> > > >>> >> >>>>>>>>>>> >> > ==20833== by 0x431BA6: wtpWorker
>> >> >> > > >>> >> >>>>>>>>>>> >> > (wtp.c:389)
>> >> >> > > >>> >> >>>>>>>>>>> >> > ==20833== by 0x3A044079D0:
>> start_thread
>> >> >> > > >>> >> >>>>>>>>>>> >> > (in
>> >> >> > > /lib64/
>> >> >> > > >>> >> >>>>>>>>>>> libpthread-2.12.so
>> >> >> > > >>> >> >>>>>>>>>>> >> )
>> >> >> > > >>> >> >>>>>>>>>>> >> > ==20833== If you believe this happened
>> as
>> >> >> > > >>> >> >>>>>>>>>>> >> > a
>> >> >> > > result of
>> >> >> > > >>> a
>> >> >> > > >>> >> stack
>> >> >> > > >>> >> >>>>>>>>>>> >> > ==20833== overflow in your program's
>> main
>> >> >> thread
>> >> >> > > >>> >> (unlikely but
>> >> >> > > >>> >> >>>>>>>>>>> >> > ==20833== possible), you can try to
>> >> >> > > >>> >> >>>>>>>>>>> >> > increase
>> >> >> the
>> >> >> > > size
>> >> >> > > >>> of
>> >> >> > > >>> >> the
>> >> >> > > >>> >> >>>>>>>>>>> >> > ==20833== main thread stack using the
>> >> >> > > >>> --main-stacksize=
>> >> >> > > >>> >> flag.
>> >> >> > > >>> >> >>>>>>>>>>> >> > ==20833== The main thread stack size
>> used
>> >> >> > > >>> >> >>>>>>>>>>> >> > in
>> >> >> > this
>> >> >> > > run
>> >> >> > > >>> >> was 10485760.
>> >> >> > > >>> >> >>>>>>>>>>> >> > ==20833==
>> >> >> > > >>> >> >>>>>>>>>>> >> > ==20833== HEAP SUMMARY:
>> >> >> > > >>> >> >>>>>>>>>>> >> > ==20833== in use at exit: 44,605,326
>> >> >> > > >>> >> >>>>>>>>>>> >> > bytes
>> >> >> in
>> >> >> > > 5,019
>> >> >> > > >>> >> blocks
>> >> >> > > >>> >> >>>>>>>>>>> >> > ==20833== total heap usage: 10,310
>> >> >> > > >>> >> >>>>>>>>>>> >> > allocs,
>> >> >> > 5,291
>> >> >> > > >>> frees,
>> >> >> > > >>> >> 78,621,963
>> >> >> > > >>> >> >>>>>>>>>>> >> bytes
>> >> >> > > >>> >> >>>>>>>>>>> >> > allocated
>> >> >> > > >>> >> >>>>>>>>>>> >> > ==20833==
>> >> >> > > >>> >> >>>>>>>>>>> >> > ==20833== LEAK SUMMARY:
>> >> >> > > >>> >> >>>>>>>>>>> >> > ==20833== definitely lost: 516 bytes
>> in
>> >> >> > > >>> >> >>>>>>>>>>> >> > 12
>> >> >> > > blocks
>> >> >> > > >>> >> >>>>>>>>>>> >> > ==20833== indirectly lost: 0 bytes in
>> 0
>> >> >> blocks
>> >> >> > > >>> >> >>>>>>>>>>> >> > ==20833== possibly lost: 33,559,553
>> >> >> > > >>> >> >>>>>>>>>>> >> > bytes
>> >> >> in
>> >> >> > > 17
>> >> >> > > >>> >> blocks
>> >> >> > > >>> >> >>>>>>>>>>> >> > ==20833== still reachable: 11,045,257
>> >> >> > > >>> >> >>>>>>>>>>> >> > bytes
>> >> >> in
>> >> >> > > 4,990
>> >> >> > > >>> >> blocks
>> >> >> > > >>> >> >>>>>>>>>>> >> > ==20833== suppressed: 0 bytes in
>> 0
>> >> >> blocks
>> >> >> > > >>> >> >>>>>>>>>>> >> > ==20833== Rerun with --leak-check=full to
>> >> >> > > >>> >> >>>>>>>>>>> >> > see
>> >> >> > > details
>> >> >> > > >>> of
>> >> >> > > >>> >> leaked memory
>> >> >> > > >>> >> >>>>>>>>>>> >> > ==20833==
>> >> >> > > >>> >> >>>>>>>>>>> >> > ==20833== For counts of detected and
>> >> >> > > >>> >> >>>>>>>>>>> >> > suppressed
>> >> >> > > errors,
>> >> >> > > >>> >> rerun with: -v
>> >> >> > > >>> >> >>>>>>>>>>> >> > ==20833== Use --track-origins=yes to see
>> >> >> > > >>> >> >>>>>>>>>>> >> > where
>> >> >> > > >>> >> uninitialised values
>> >> >> > > >>> >> >>>>>>>>>>> come
>> >> >> > > >>> >> >>>>>>>>>>> >> > from
>> >> >> > > >>> >> >>>>>>>>>>> >> > ==20833== ERROR SUMMARY: 11 errors from 3
>> >> >> > contexts
>> >> >> > > >>> >> (suppressed: 100
>> >> >> > > >>> >> >>>>>>>>>>> from
>> >> >> > > >>> >> >>>>>>>>>>> >> 9)
>> >> >> > > >>> >> >>>>>>>>>>> >> > 已杀死
>> >> >> > > >>> >> >>>>>>>>>>> >> >
>> >> >> > > >>> >> >>>>>>>>>>> >> > 2015-03-20 23:29 GMT+08:00
>> singh.janmejay <
>> >> >> > > >>> >> singh.janme...@gmail.com>:
>> >> >> > > >>> >> >>>>>>>>>>> >> >
>> >> >> > > >>> >> >>>>>>>>>>> >> > > Can you please build with debug symbols
>> >> >> > > >>> >> >>>>>>>>>>> >> > > and
>> >> >> > > repeat
>> >> >> > > >>> the
>> >> >> > > >>> >> valgrind run?
>> >> >> > > >>> >> >>>>>>>>>>> >> > >
>> >> >> > > >>> >> >>>>>>>>>>> >> > > --
>> >> >> > > >>> >> >>>>>>>>>>> >> > > Regards,
>> >> >> > > >>> >> >>>>>>>>>>> >> > > Janmejay
>> >> >> > > >>> >> >>>>>>>>>>> >> > >
>> >> >> > > >>> >> >>>>>>>>>>> >> > > PS: Please blame the typos in this mail
>> >> >> > > >>> >> >>>>>>>>>>> >> > > on my
>> >> >> > > phone's
>> >> >> > > >>> >> uncivilized
>> >> >> > > >>> >> >>>>>>>>>>> soft
>> >> >> > > >>> >> >>>>>>>>>>> >> > > keyboard sporting it's
>> >> >> > > >>> >> >>>>>>>>>>> >> > > not-so-smart-assist
>> >> >> > > >>> technology.
>> >> >> > > >>> >> >>>>>>>>>>> >> > >
>> >> >> > > >>> >> >>>>>>>>>>> >> > > On Mar 20, 2015 6:03 PM, "chenlin rao"
>> <
>> >> >> > > >>> >> rao.chen...@gmail.com>
>> >> >> > > >>> >> >>>>>>>>>>> wrote:
>> >> >> > > >>> >> >>>>>>>>>>> >> > >
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > btw: if I change
>> omelasticsearch/omfwd
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > to
>> >> >> > > omfile,
>> >> >> > > >>> >> rsyslogd would
>> >> >> > > >>> >> >>>>>>>>>>> be
>> >> >> > > >>> >> >>>>>>>>>>> >> > > fine...
>> >> >> > > >>> >> >>>>>>>>>>> >> > > >
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > 2015-03-20 20:13 GMT+08:00 chenlin
>> rao
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > <
>> >> >> > > >>> >> rao.chen...@gmail.com>:
>> >> >> > > >>> >> >>>>>>>>>>> >> > > >
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > 3498.767218405:main Q[DA]:Reg/w0:
>> >> >> > > rainerscript:
>> >> >> > > >>> var
>> >> >> > > >>> >> 200:!msg:
>> >> >> > > >>> >> >>>>>>>>>>> '[ {
>> >> >> > > >>> >> >>>>>>>>>>> >> > > "uid":
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > "1941604034", "request_header":
>> >> >> > > >>> >> >>>>>>>>>>> >> > >
>> "{\"Accept-Encoding\":\"gzip,deflate\"}",
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > "network_type": "wifi", "end_time":
>> >> >> > > >>> >> "1426836307406", "dns_ip":
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > "218.15.203.34,192.168.1.1",
>> >> >> > "response_code":
>> >> >> > > >>> "200",
>> >> >> > > >>> >> >>>>>>>>>>> >> "response_data":
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > "{}",
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > "start_time": "1426836307328",
>> "act":
>> >> >> > > >>> >> "net_fatal_error", "type":
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > "net_fatal_error", "request_url":
>> >> >> > "https:\/\/
>> >> >> > > >>> >> api.weibo.cn
>> >> >> > > >>> >> >>>>>>>>>>> >> > > >
>> >> >> > > >>> >> >>>>>>>>>>> >> > >
>> >> >> > > >>> >> >>>>>>>>>>> >> >
>> >> >> > > >>> >> >>>>>>>>>>> >>
>> >> >> > > >>> >> >>>>>>>>>>>
>> >> >> > > >>> >>
>> >> >> > > >>>
>> >> >> > >
>> >> >> >
>> >> >>
>> >> >>
>> \/2\/client\/url_list?accuracy_level=0&c=android&i=9eef7ba&s=0088f6a4&ua=Xiaomi-MI%202S__weibo__5.0.0__android__android4.4.4&wm=5311_4002&v_f=2&from=1050095010&gsid=4uzH00573m6YZU5oVhYra896c0y&lang=zh_CN&skin=default&oldwm=5311_4002",
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > "response_status_line": "HTTP\/1.1
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > 200
>> >> >> OK"
>> >> >> > > }, {
>> >> >> > > >>> >> "uid":
>> >> >> > > >>> >> >>>>>>>>>>> >> "1941604034",
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > "request_header":
>> >> >> > > >>> >> "{\"Accept-Encoding\":\"gzip,deflate\"}",
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > "network_type":
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > "wifi", "end_time":
>> "1426836320563",
>> >> >> > > "dns_ip":
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > "218.15.203.34,192.168.1.1",
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > "response_code": "200",
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > "response_data":
>> >> >> > > "{}",
>> >> >> > > >>> >> "start_time":
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > "1426836320505", "act":
>> >> >> "net_fatal_error",
>> >> >> > > >>> "type":
>> >> >> > > >>> >> >>>>>>>>>>> >> "net_fatal_error",
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > "request_url":
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > "https:\/\/api.weibo.cn
>> >> >> > > >>> >> >>>>>>>>>>> >> > > >
>> >> >> > > >>> >> >>>>>>>>>>> >> > >
>> >> >> > > >>> >> >>>>>>>>>>> >> >
>> >> >> > > >>> >> >>>>>>>>>>> >>
>> >> >> > > >>> >> >>>>>>>>>>>
>> >> >> > > >>> >>
>> >> >> > > >>>
>> >> >> > >
>> >> >> >
>> >> >>
>> >> >>
>> \/2\/client\/url_list?accuracy_level=0&c=android&i=9eef7ba&s=0088f6a4&ua=Xiaomi-MI%202S__weibo__5.0.0__android__android4.4.4&wm=5311_4002&v_f=2&from=1050095010&gsid=4uzH00573m6YZU5oVhYra896c0y&lang=zh_CN&skin=default&oldwm=5311_4002",
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > "response_status_line": "HTTP\/1.1
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > 200
>> >> >> OK"
>> >> >> > }
>> >> >> > > ]'
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > 3498.767338611:main Q[DA]:Reg/w0:
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > eval
>> >> >> expr
>> >> >> > > >>> >> 0x62e63a0, return
>> >> >> > > >>> >> >>>>>>>>>>> >> > datatype
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > 'J'
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > 3498.770732236:main Q[DA]:Reg/w0:
>> >> >> > ACTION
>> >> >> > > 17
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >
>> >> >> [builtin:omfwd:action(type="builtin:omfwd"
>> >> >> > > ...)]
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > 3498.770963455:main Q[DA]:Reg/w0:
>> >> >> executing
>> >> >> > > >>> action
>> >> >> > > >>> >> 17
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > 3498.771052511:main Q[DA]:Reg/w0:
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > Called
>> >> >> > > action,
>> >> >> > > >>> >> logging to
>> >> >> > > >>> >> >>>>>>>>>>> >> > > builtin:omfwd
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > ==30138== Thread 3:
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > ==30138== Invalid read of size 4
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > ==30138== at 0x4E3E3A0:
>> >> >> > > pthread_mutex_lock (in
>> >> >> > > >>> >> /lib64/
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > libpthread-2.12.so)
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > ==30138== by 0x14228D:
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > qqueueEnqMsg
>> >> >> (in
>> >> >> > > >>> >> /sbin/rsyslogd)
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > ==30138== by 0x14BB5B: ??? (in
>> >> >> > > /sbin/rsyslogd)
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > ==30138== by 0x143C4C: ??? (in
>> >> >> > > /sbin/rsyslogd)
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > ==30138== by 0x143DAC: ??? (in
>> >> >> > > /sbin/rsyslogd)
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > ==30138== by 0x143BB9: ??? (in
>> >> >> > > /sbin/rsyslogd)
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > ==30138== by 0x1442CB: ??? (in
>> >> >> > > /sbin/rsyslogd)
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > ==30138== by 0x15336B: ??? (in
>> >> >> > > /sbin/rsyslogd)
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > ==30138== by 0x143112: ??? (in
>> >> >> > > /sbin/rsyslogd)
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > ==30138== by 0x13D671: wtiWorker
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > (in
>> >> >> > > >>> >> /sbin/rsyslogd)
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > ==30138== by 0x13D1C1: ??? (in
>> >> >> > > /sbin/rsyslogd)
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > ==30138== by 0x4E3C9D0:
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > start_thread
>> >> >> (in
>> >> >> > > >>> /lib64/
>> >> >> > > >>> >> >>>>>>>>>>> >> > libpthread-2.12.so)
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > ==30138== Address 0x10 is not
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > stack'd,
>> >> >> > > malloc'd
>> >> >> > > >>> or
>> >> >> > > >>> >> (recently)
>> >> >> > > >>> >> >>>>>>>>>>> >> free'd
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > ==30138==
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > ==30138==
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > ==30138== Process terminating with
>> >> >> default
>> >> >> > > action
>> >> >> > > >>> >> of signal 11
>> >> >> > > >>> >> >>>>>>>>>>> >> > > (SIGSEGV)
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > ==30138== Access not within mapped
>> >> >> region
>> >> >> > at
>> >> >> > > >>> >> address 0x10
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > ==30138== at 0x4E3E3A0:
>> >> >> > > pthread_mutex_lock (in
>> >> >> > > >>> >> /lib64/
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > libpthread-2.12.so)
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > ==30138== by 0x14228D:
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > qqueueEnqMsg
>> >> >> (in
>> >> >> > > >>> >> /sbin/rsyslogd)
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > ==30138== by 0x14BB5B: ??? (in
>> >> >> > > /sbin/rsyslogd)
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > ==30138== by 0x143C4C: ??? (in
>> >> >> > > /sbin/rsyslogd)
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > ==30138== by 0x143DAC: ??? (in
>> >> >> > > /sbin/rsyslogd)
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > ==30138== by 0x143BB9: ??? (in
>> >> >> > > /sbin/rsyslogd)
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > ==30138== by 0x1442CB: ??? (in
>> >> >> > > /sbin/rsyslogd)
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > ==30138== by 0x15336B: ??? (in
>> >> >> > > /sbin/rsyslogd)
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > ==30138== by 0x143112: ??? (in
>> >> >> > > /sbin/rsyslogd)
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > ==30138== by 0x13D671: wtiWorker
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > (in
>> >> >> > > >>> >> /sbin/rsyslogd)
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > ==30138== by 0x13D1C1: ??? (in
>> >> >> > > /sbin/rsyslogd)
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > ==30138== by 0x4E3C9D0:
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > start_thread
>> >> >> (in
>> >> >> > > >>> /lib64/
>> >> >> > > >>> >> >>>>>>>>>>> >> > libpthread-2.12.so)
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > ==30138== If you believe this
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > happened
>> >> >> as
>> >> >> > a
>> >> >> > > >>> result
>> >> >> > > >>> >> of a stack
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > ==30138== overflow in your
>> program's
>> >> >> main
>> >> >> > > thread
>> >> >> > > >>> >> (unlikely but
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > ==30138== possible), you can try
>> to
>> >> >> > > increase the
>> >> >> > > >>> >> size of the
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > ==30138== main thread stack using
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > the
>> >> >> > > >>> >> --main-stacksize= flag.
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > ==30138== The main thread stack
>> size
>> >> >> used
>> >> >> > in
>> >> >> > > >>> this
>> >> >> > > >>> >> run was
>> >> >> > > >>> >> >>>>>>>>>>> >> 10485760.
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > ==30138==
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > ==30138== HEAP SUMMARY:
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > ==30138== in use at exit:
>> >> >> 1,078,127,378
>> >> >> > > bytes
>> >> >> > > >>> >> in 291,997
>> >> >> > > >>> >> >>>>>>>>>>> blocks
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > ==30138== total heap usage:
>> 614,872
>> >> >> > allocs,
>> >> >> > > >>> >> 322,875 frees,
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > 1,586,744,205
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > bytes allocated
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > ==30138==
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > ==30138== LEAK SUMMARY:
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > ==30138== definitely lost:
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > 33,555,027
>> >> >> > > bytes in
>> >> >> > > >>> >> 13 blocks
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > ==30138== indirectly lost: 0
>> bytes
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > in
>> >> >> 0
>> >> >> > > blocks
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > ==30138== possibly lost: 5,760
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > bytes
>> >> >> > in
>> >> >> > > 18
>> >> >> > > >>> >> blocks
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > ==30138== still reachable:
>> >> >> 1,044,566,591
>> >> >> > > bytes
>> >> >> > > >>> >> in 291,966
>> >> >> > > >>> >> >>>>>>>>>>> blocks
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > ==30138== suppressed: 0
>> bytes
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > in
>> >> >> 0
>> >> >> > > blocks
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > ==30138== Rerun with
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > --leak-check=full to
>> >> >> > see
>> >> >> > > >>> >> details of leaked
>> >> >> > > >>> >> >>>>>>>>>>> >> > memory
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > ==30138==
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > ==30138== For counts of detected
>> and
>> >> >> > > suppressed
>> >> >> > > >>> >> errors, rerun
>> >> >> > > >>> >> >>>>>>>>>>> with:
>> >> >> > > >>> >> >>>>>>>>>>> >> > -v
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > ==30138== Use --track-origins=yes
>> to
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > see
>> >> >> > > where
>> >> >> > > >>> >> uninitialised
>> >> >> > > >>> >> >>>>>>>>>>> values
>> >> >> > > >>> >> >>>>>>>>>>> >> > > come
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > from
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > ==30138== ERROR SUMMARY: 1809
>> errors
>> >> >> from 5
>> >> >> > > >>> contexts
>> >> >> > > >>> >> >>>>>>>>>>> (suppressed:
>> >> >> > > >>> >> >>>>>>>>>>> >> 131
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > from
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > 9)
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > Killed
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > 2015-03-20 8:50 GMT+08:00
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > > singh.janmejay
>> >> >> <
>> >> >> > > >>> >> >>>>>>>>>>> singh.janme...@gmail.com
>> >> >> > > >>> >> >>>>>>>>>>> >> >:
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> Can you please run this with
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> valgrind
>> >> >> and
>> >> >> > > share
>> >> >> > > >>> >> its output on
>> >> >> > > >>> >> >>>>>>>>>>> >> crash?
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >>
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> --
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> Regards,
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> Janmejay
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >>
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> PS: Please blame the typos in this
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> mail
>> >> >> on
>> >> >> > > my
>> >> >> > > >>> >> phone's
>> >> >> > > >>> >> >>>>>>>>>>> uncivilized
>> >> >> > > >>> >> >>>>>>>>>>> >> > soft
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> keyboard sporting it's
>> >> >> not-so-smart-assist
>> >> >> > > >>> >> technology.
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >>
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> On Mar 19, 2015 11:10 PM, "chenlin
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> rao"
>> >> >> <
>> >> >> > > >>> >> rao.chen...@gmail.com
>> >> >> > > >>> >> >>>>>>>>>>> >
>> >> >> > > >>> >> >>>>>>>>>>> >> > > wrote:
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >>
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> > Hello everyone.
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> >
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> > I just learnt a foreach syntax
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> > from
>> >> >> > > >>> >> >>>>>>>>>>> >> > > `src/tests/json_array_looping.sh`,
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> so I
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> > try to parse my logdata(yes,
>> long
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> > json
>> >> >> > > array
>> >> >> > > >>> in
>> >> >> > > >>> >> msg) as
>> >> >> > > >>> >> >>>>>>>>>>> follow:
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> >
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> > ```
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> >
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> > $MaxMessageSize 256k
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> >
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> > template( name="local6JsonArray"
>> >> >> > > type="string"
>> >> >> > > >>> >> >>>>>>>>>>> >> > string="%$.line%\n" )
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> >
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> > Ruleset(
>> >> >> name="forwardRuleSetJsonArray"
>> >> >> > )
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> >
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> > {
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> >
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> > action( type="mmjsonparse" )
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> >
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> > foreach ($.line in $!msg)
>> do {
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> >
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> > action ( type="omfile"
>> >> >> > > >>> >> file="/data0/logfile"
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> > template="local6JsonArray")
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> > }
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> > }
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> > ```
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> >
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> > But I always got segment fault
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> > after
>> >> >> > > restarted
>> >> >> > > >>> >> few minutes.
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> >
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> > Before the rsyslogd died , I can
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> > watch
>> >> >> > > some
>> >> >> > > >>> >> correct lines in
>> >> >> > > >>> >> >>>>>>>>>>> the
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> > "/data0/logfile".
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> >
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> > While running `rsyslogd -dn`, I
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> > watch
>> >> >> > the
>> >> >> > > last
>> >> >> > > >>> >> five lines
>> >> >> > > >>> >> >>>>>>>>>>> were:
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> >
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> > ```
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> >
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> > 3295.820218609:main Q:Reg/w0 :
>> >> >> > > FOREACH
>> >> >> > > >>> >> .line IN
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> >
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> > 3295.820227632:main Q:Reg/w0 :
>> >> >> > var
>> >> >> > > >>> '!msg'
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> >
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> > 3295.820240109:main Q:Reg/w0 :
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> > eval
>> >> >> > expr
>> >> >> > > >>> >> 0x7fe2044fefa0,
>> >> >> > > >>> >> >>>>>>>>>>> type
>> >> >> > > >>> >> >>>>>>>>>>> >> > > 'V[86]'
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> >
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> > 3295.821747011:main Q:Reg/w0 :
>> >> >> > > rainerscript:
>> >> >> > > >>> >> var 200:!msg:
>> >> >> > > >>> >> >>>>>>>>>>> >> > > '{"msg":[{
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> >
>> >> >> > > "end_time":"142...longmsghere....\"verieval
>> >> >> > > >>> expr
>> >> >> > > >>> >> >>>>>>>>>>> 0x7fe2044fefa0,
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > return
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> > datatype 'J'
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> > ```
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> >
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> > The length of such line is about
>> >> >> > 20k~30k,
>> >> >> > > far
>> >> >> > > >>> >> away below my
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> > $MaxMessageSize.
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> >
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> > So, why rsyslogd segment fault?
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> >
>> >> >> > > >>> _______________________________________________
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> > rsyslog mailing list
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> >
>> >> >> > > >>> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> >
>> >> >> > > http://www.rsyslog.com/professional-services/
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> > What's up with rsyslog? Follow
>> >> >> > > >>> >> https://twitter.com/rgerhards
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> > NOTE WELL: This is a PUBLIC
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> > mailing
>> >> >> > list,
>> >> >> > > >>> posts
>> >> >> > > >>> >> are ARCHIVED
>> >> >> > > >>> >> >>>>>>>>>>> by
>> >> >> > > >>> >> >>>>>>>>>>> >> a
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > myriad
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> > of sites beyond our control.
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> > PLEASE
>> >> >> > > >>> UNSUBSCRIBE
>> >> >> > > >>> >> and DO NOT
>> >> >> > > >>> >> >>>>>>>>>>> POST
>> >> >> > > >>> >> >>>>>>>>>>> >> if
>> >> >> > > >>> >> >>>>>>>>>>> >> > > you
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> > DON'T LIKE THAT.
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> >
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >>
>> >> >> > > _______________________________________________
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> rsyslog mailing list
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >>
>> >> >> > > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >>
>> >> >> > > http://www.rsyslog.com/professional-services/
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> What's up with rsyslog? Follow
>> >> >> > > >>> >> https://twitter.com/rgerhards
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> NOTE WELL: This is a PUBLIC
>> mailing
>> >> >> list,
>> >> >> > > posts
>> >> >> > > >>> >> are ARCHIVED
>> >> >> > > >>> >> >>>>>>>>>>> by a
>> >> >> > > >>> >> >>>>>>>>>>> >> > > myriad
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> of sites beyond our control.
>> PLEASE
>> >> >> > > UNSUBSCRIBE
>> >> >> > > >>> >> and DO NOT
>> >> >> > > >>> >> >>>>>>>>>>> POST if
>> >> >> > > >>> >> >>>>>>>>>>> >> > you
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >> DON'T LIKE THAT.
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >>
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > >
>> >> >> > > >>> >> >>>>>>>>>>> >> > > >
>> >> >> > _______________________________________________
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > rsyslog mailing list
>> >> >> > > >>> >> >>>>>>>>>>> >> > > >
>> >> >> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >> >> > > >>> >> >>>>>>>>>>> >> > > >
>> >> >> > http://www.rsyslog.com/professional-services/
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > What's up with rsyslog? Follow
>> >> >> > > >>> >> https://twitter.com/rgerhards
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > NOTE WELL: This is a PUBLIC mailing
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > list,
>> >> >> > > posts are
>> >> >> > > >>> >> ARCHIVED by a
>> >> >> > > >>> >> >>>>>>>>>>> >> > myriad
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > of sites beyond our control. PLEASE
>> >> >> > > UNSUBSCRIBE and
>> >> >> > > >>> >> DO NOT POST if
>> >> >> > > >>> >> >>>>>>>>>>> >> you
>> >> >> > > >>> >> >>>>>>>>>>> >> > > > DON'T LIKE THAT.
>> >> >> > > >>> >> >>>>>>>>>>> >> > > >
>> >> >> > > >>> >> >>>>>>>>>>> >> > >
>> >> >> _______________________________________________
>> >> >> > > >>> >> >>>>>>>>>>> >> > > rsyslog mailing list
>> >> >> > > >>> >> >>>>>>>>>>> >> > >
>> >> >> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >> >> > > >>> >> >>>>>>>>>>> >> > >
>> >> >> http://www.rsyslog.com/professional-services/
>> >> >> > > >>> >> >>>>>>>>>>> >> > > What's up with rsyslog? Follow
>> >> >> > > >>> >> https://twitter.com/rgerhards
>> >> >> > > >>> >> >>>>>>>>>>> >> > > NOTE WELL: This is a PUBLIC mailing
>> list,
>> >> >> posts
>> >> >> > > are
>> >> >> > > >>> >> ARCHIVED by a
>> >> >> > > >>> >> >>>>>>>>>>> >> myriad
>> >> >> > > >>> >> >>>>>>>>>>> >> > > of sites beyond our control. PLEASE
>> >> >> UNSUBSCRIBE
>> >> >> > > and
>> >> >> > > >>> DO
>> >> >> > > >>> >> NOT POST if
>> >> >> > > >>> >> >>>>>>>>>>> you
>> >> >> > > >>> >> >>>>>>>>>>> >> > > DON'T LIKE THAT.
>> >> >> > > >>> >> >>>>>>>>>>> >> > >
>> >> >> > > >>> >> >>>>>>>>>>> >> >
>> >> >> > > >>> >> >>>>>>>>>>> >> >
>> _______________________________________________
>> >> >> > > >>> >> >>>>>>>>>>> >> > rsyslog mailing list
>> >> >> > > >>> >> >>>>>>>>>>> >> >
>> >> >> > http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >> >> > > >>> >> >>>>>>>>>>> >> >
>> >> >> > > >>> >> >>>>>>>>>>> >> >
>> http://www.rsyslog.com/professional-services/
>> >> >> > > >>> >> >>>>>>>>>>> >> > What's up with rsyslog? Follow
>> >> >> > > >>> >> https://twitter.com/rgerhards
>> >> >> > > >>> >> >>>>>>>>>>> >> > NOTE WELL: This is a PUBLIC mailing list,
>> >> >> > > >>> >> >>>>>>>>>>> >> > posts
>> >> >> > are
>> >> >> > > >>> >> ARCHIVED by a
>> >> >> > > >>> >> >>>>>>>>>>> myriad
>> >> >> > > >>> >> >>>>>>>>>>> >> > of sites beyond our control. PLEASE
>> >> >> > > >>> >> >>>>>>>>>>> >> > UNSUBSCRIBE
>> >> >> > > and DO
>> >> >> > > >>> >> NOT POST if you
>> >> >> > > >>> >> >>>>>>>>>>> >> > DON'T LIKE THAT.
>> >> >> > > >>> >> >>>>>>>>>>> >>
>> >> >> > > >>> >> >>>>>>>>>>> >>
>> _______________________________________________
>> >> >> > > >>> >> >>>>>>>>>>> >> rsyslog mailing list
>> >> >> > > >>> >> >>>>>>>>>>> >>
>> >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >> >> > > >>> >> >>>>>>>>>>> >>
>> http://www.rsyslog.com/professional-services/
>> >> >> > > >>> >> >>>>>>>>>>> >> What's up with rsyslog? Follow
>> >> >> > > >>> >> https://twitter.com/rgerhards
>> >> >> > > >>> >> >>>>>>>>>>> >> NOTE WELL: This is a PUBLIC mailing list,
>> >> >> > > >>> >> >>>>>>>>>>> >> posts
>> >> >> are
>> >> >> > > >>> >> ARCHIVED by a myriad
>> >> >> > > >>> >> >>>>>>>>>>> >> of sites beyond our control. PLEASE
>> >> >> > > >>> >> >>>>>>>>>>> >> UNSUBSCRIBE
>> >> >> and
>> >> >> > > DO
>> >> >> > > >>> NOT
>> >> >> > > >>> >> POST if you
>> >> >> > > >>> >> >>>>>>>>>>> >> DON'T LIKE THAT.
>> >> >> > > >>> >> >>>>>>>>>>> >>
>> >> >> > > >>> >> >>>>>>>>>>> >
>> >> >> > > >>> >> >>>>>>>>>>> >
>> _______________________________________________
>> >> >> > > >>> >> >>>>>>>>>>> > rsyslog mailing list
>> >> >> > > >>> >> >>>>>>>>>>> >
>> >> >> > > >>> >> >>>>>>>>>>> >
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >> >> > > >>> >> >>>>>>>>>>> >
>> http://www.rsyslog.com/professional-services/
>> >> >> > > >>> >> >>>>>>>>>>> > What's up with rsyslog? Follow
>> >> >> > > >>> https://twitter.com/rgerhards
>> >> >> > > >>> >> >>>>>>>>>>> > NOTE WELL: This is a PUBLIC mailing list,
>> >> >> > > >>> >> >>>>>>>>>>> > posts
>> >> >> are
>> >> >> > > >>> ARCHIVED
>> >> >> > > >>> >> by a myriad
>> >> >> > > >>> >> >>>>>>>>>>> of sites beyond our control. PLEASE
>> UNSUBSCRIBE
>> >> >> > > >>> >> >>>>>>>>>>> and
>> >> >> DO
>> >> >> > > NOT
>> >> >> > > >>> >> POST if you
>> >> >> > > >>> >> >>>>>>>>>>> DON'T LIKE THAT.
>> >> >> > > >>> >> >>>>>>>>>>>
>> _______________________________________________
>> >> >> > > >>> >> >>>>>>>>>>> rsyslog mailing list
>> >> >> > > >>> >> >>>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>>>
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >> >> > > >>> >> >>>>>>>>>>> http://www.rsyslog.com/professional-services/
>> >> >> > > >>> >> >>>>>>>>>>> What's up with rsyslog? Follow
>> >> >> > > >>> https://twitter.com/rgerhards
>> >> >> > > >>> >> >>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list,
>> posts
>> >> >> > > >>> >> >>>>>>>>>>> are
>> >> >> > > ARCHIVED
>> >> >> > > >>> >> by a myriad
>> >> >> > > >>> >> >>>>>>>>>>> of sites beyond our control. PLEASE
>> UNSUBSCRIBE
>> >> >> > > >>> >> >>>>>>>>>>> and
>> >> >> DO
>> >> >> > > NOT
>> >> >> > > >>> >> POST if you
>> >> >> > > >>> >> >>>>>>>>>>> DON'T LIKE THAT.
>> >> >> > > >>> >> >>>>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>>> _______________________________________________
>> >> >> > > >>> >> >>>>>>>>>> rsyslog mailing list
>> >> >> > > >>> >> >>>>>>>>>>
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >> >> > > >>> >> >>>>>>>>>> http://www.rsyslog.com/professional-services/
>> >> >> > > >>> >> >>>>>>>>>> What's up with rsyslog? Follow
>> >> >> > > https://twitter.com/rgerhards
>> >> >> > > >>> >> >>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts
>> >> >> > > >>> >> >>>>>>>>>> are
>> >> >> > > ARCHIVED
>> >> >> > > >>> by
>> >> >> > > >>> >> a myriad of sites beyond our control. PLEASE UNSUBSCRIBE
>> and
>> >> >> > > >>> >> DO
>> >> >> > NOT
>> >> >> > > >>> POST if
>> >> >> > > >>> >> you DON'T LIKE THAT.
>> >> >> > > >>> >> >>>>>>>>>
>> >> >> > > >>> >> >>>>>>>>> _______________________________________________
>> >> >> > > >>> >> >>>>>>>>> rsyslog mailing list
>> >> >> > > >>> >> >>>>>>>>>
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >> >> > > >>> >> >>>>>>>>> http://www.rsyslog.com/professional-services/
>> >> >> > > >>> >> >>>>>>>>> What's up with rsyslog? Follow
>> >> >> > > https://twitter.com/rgerhards
>> >> >> > > >>> >> >>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts
>> >> >> > > >>> >> >>>>>>>>> are
>> >> >> > > ARCHIVED
>> >> >> > > >>> by
>> >> >> > > >>> >> a myriad of sites beyond our control. PLEASE UNSUBSCRIBE
>> and
>> >> >> > > >>> >> DO
>> >> >> > NOT
>> >> >> > > >>> POST if
>> >> >> > > >>> >> you DON'T LIKE THAT.
>> >> >> > > >>> >> >>>>>>>>
>> >> >> > > >>> >> >>>>>>>>
>> >> >> > > >>> >> >>>>>>>>
>> >> >> > > >>> >> >>>>>>>> --
>> >> >> > > >>> >> >>>>>>>> Regards,
>> >> >> > > >>> >> >>>>>>>> Janmejay
>> >> >> > > >>> >> >>>>>>>> http://codehunk.wordpress.com
>> >> >> > > >>> >> >>>>>>>> _______________________________________________
>> >> >> > > >>> >> >>>>>>>> rsyslog mailing list
>> >> >> > > >>> >> >>>>>>>>
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >> >> > > >>> >> >>>>>>>> http://www.rsyslog.com/professional-services/
>> >> >> > > >>> >> >>>>>>>> What's up with rsyslog? Follow
>> >> >> > > https://twitter.com/rgerhards
>> >> >> > > >>> >> >>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts
>> are
>> >> >> > > ARCHIVED
>> >> >> > > >>> by a
>> >> >> > > >>> >> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and
>> >> >> > > >>> >> DO
>> >> >> NOT
>> >> >> > > POST
>> >> >> > > >>> if
>> >> >> > > >>> >> you DON'T LIKE THAT.
>> >> >> > > >>> >> >>>>>>> _______________________________________________
>> >> >> > > >>> >> >>>>>>> rsyslog mailing list
>> >> >> > > >>> >> >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >> >> > > >>> >> >>>>>>> http://www.rsyslog.com/professional-services/
>> >> >> > > >>> >> >>>>>>> What's up with rsyslog? Follow
>> >> >> > > https://twitter.com/rgerhards
>> >> >> > > >>> >> >>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts
>> are
>> >> >> > > ARCHIVED by
>> >> >> > > >>> a
>> >> >> > > >>> >> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and
>> >> >> > > >>> >> DO
>> >> >> NOT
>> >> >> > > POST
>> >> >> > > >>> if
>> >> >> > > >>> >> you DON'T LIKE THAT.
>> >> >> > > >>> >> >>>>>>
>> >> >> > > >>> >> >>>>>>
>> >> >> > > >>> >> >>>>>>
>> >> >> > > >>> >> >>>>>> --
>> >> >> > > >>> >> >>>>>> Regards,
>> >> >> > > >>> >> >>>>>> Janmejay
>> >> >> > > >>> >> >>>>>> http://codehunk.wordpress.com
>> >> >> > > >>> >> >>>>>
>> >> >> > > >>> >> >>>>>
>> >> >> > > >>> >> >>>>>
>> >> >> > > >>> >> >>>>> --
>> >> >> > > >>> >> >>>>> Regards,
>> >> >> > > >>> >> >>>>> Janmejay
>> >> >> > > >>> >> >>>>> http://codehunk.wordpress.com
>> >> >> > > >>> >> >>>>> _______________________________________________
>> >> >> > > >>> >> >>>>> rsyslog mailing list
>> >> >> > > >>> >> >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >> >> > > >>> >> >>>>> http://www.rsyslog.com/professional-services/
>> >> >> > > >>> >> >>>>> What's up with rsyslog? Follow
>> >> >> > https://twitter.com/rgerhards
>> >> >> > > >>> >> >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are
>> >> >> ARCHIVED
>> >> >> > > by a
>> >> >> > > >>> >> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and
>> >> >> > > >>> >> DO
>> >> >> NOT
>> >> >> > > POST
>> >> >> > > >>> if
>> >> >> > > >>> >> you DON'T LIKE THAT.
>> >> >> > > >>> >> >>>> _______________________________________________
>> >> >> > > >>> >> >>>> rsyslog mailing list
>> >> >> > > >>> >> >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >> >> > > >>> >> >>>> http://www.rsyslog.com/professional-services/
>> >> >> > > >>> >> >>>> What's up with rsyslog? Follow
>> >> >> https://twitter.com/rgerhards
>> >> >> > > >>> >> >>>> NOTE WELL: This is a PUBLIC mailing list, posts are
>> >> >> ARCHIVED
>> >> >> > > by a
>> >> >> > > >>> >> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and
>> >> >> > > >>> >> DO
>> >> >> NOT
>> >> >> > > POST
>> >> >> > > >>> if
>> >> >> > > >>> >> you DON'T LIKE THAT.
>> >> >> > > >>> >> >>>
>> >> >> > > >>> >> >>>
>> >> >> > > >>> >> >>>
>> >> >> > > >>> >> >>> --
>> >> >> > > >>> >> >>> Regards,
>> >> >> > > >>> >> >>> Janmejay
>> >> >> > > >>> >> >>> http://codehunk.wordpress.com
>> >> >> > > >>> >> >>
>> >> >> > > >>> >> >>
>> >> >> > > >>> >> >>
>> >> >> > > >>> >> >> --
>> >> >> > > >>> >> >> Regards,
>> >> >> > > >>> >> >> Janmejay
>> >> >> > > >>> >> >> http://codehunk.wordpress.com
>> >> >> > > >>> >> >
>> >> >> > > >>> >> >
>> >> >> > > >>> >> >
>> >> >> > > >>> >> > --
>> >> >> > > >>> >> > Regards,
>> >> >> > > >>> >> > Janmejay
>> >> >> > > >>> >> > http://codehunk.wordpress.com
>> >> >> > > >>> >>
>> >> >> > > >>> >>
>> >> >> > > >>> >>
>> >> >> > > >>> >> --
>> >> >> > > >>> >> Regards,
>> >> >> > > >>> >> Janmejay
>> >> >> > > >>> >> http://codehunk.wordpress.com
>> >> >> > > >>> >> _______________________________________________
>> >> >> > > >>> >> rsyslog mailing list
>> >> >> > > >>> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >> >> > > >>> >> http://www.rsyslog.com/professional-services/
>> >> >> > > >>> >> What's up with rsyslog? Follow
>> https://twitter.com/rgerhards
>> >> >> > > >>> >> NOTE WELL: This is a PUBLIC mailing list, posts are
>> ARCHIVED
>> >> >> > > >>> >> by
>> >> >> a
>> >> >> > > myriad
>> >> >> > > >>> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>> >> >> > > >>> >> POST
>> >> >> if
>> >> >> > > you
>> >> >> > > >>> >> DON'T LIKE THAT.
>> >> >> > > >>> >>
>> >> >> > > >>> > _______________________________________________
>> >> >> > > >>> > rsyslog mailing list
>> >> >> > > >>> > http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >> >> > > >>> > http://www.rsyslog.com/professional-services/
>> >> >> > > >>> > What's up with rsyslog? Follow
>> https://twitter.com/rgerhards
>> >> >> > > >>> > NOTE WELL: This is a PUBLIC mailing list, posts are
>> ARCHIVED
>> >> >> > > >>> > by a
>> >> >> > > myriad
>> >> >> > > >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>> POST
>> >> >> > > >>> if
>> >> >> > you
>> >> >> > > >>> DON'T LIKE THAT.
>> >> >> > > >>>
>> >> >> > > >>>
>> >> >> > > >>>
>> >> >> > > >>> --
>> >> >> > > >>> Regards,
>> >> >> > > >>> Janmejay
>> >> >> > > >>> http://codehunk.wordpress.com
>> >> >> > > >>> _______________________________________________
>> >> >> > > >>> rsyslog mailing list
>> >> >> > > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >> >> > > >>> http://www.rsyslog.com/professional-services/
>> >> >> > > >>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> >> >> > > >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
>> by
>> >> >> > > >>> a
>> >> >> > > myriad
>> >> >> > > >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>> POST
>> >> >> > > >>> if
>> >> >> > you
>> >> >> > > >>> DON'T LIKE THAT.
>> >> >> > > >>>
>> >> >> > > >> _______________________________________________
>> >> >> > > >> rsyslog mailing list
>> >> >> > > >> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >> >> > > >> http://www.rsyslog.com/professional-services/
>> >> >> > > >> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> >> >> > > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
>> by
>> >> >> > > >> a
>> >> >> > > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>> >> >> > > POST
>> >> >> if
>> >> >> > > you DON'T LIKE THAT.
>> >> >> > > >
>> >> >> > > >
>> >> >> > > >
>> >> >> > > > --
>> >> >> > > > Regards,
>> >> >> > > > Janmejay
>> >> >> > > > http://codehunk.wordpress.com
>> >> >> > >
>> >> >> > >
>> >> >> > >
>> >> >> > > --
>> >> >> > > Regards,
>> >> >> > > Janmejay
>> >> >> > > http://codehunk.wordpress.com
>> >> >> > > _______________________________________________
>> >> >> > > rsyslog mailing list
>> >> >> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >> >> > > http://www.rsyslog.com/professional-services/
>> >> >> > > What's up with rsyslog? Follow https://twitter.com/rgerhards
>> >> >> > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> >> >> myriad
>> >> >> > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
>> if
>> >> >> > > you
>> >> >> > > DON'T LIKE THAT.
>> >> >> > >
>> >> >> > _______________________________________________
>> >> >> > rsyslog mailing list
>> >> >> > http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >> >> > http://www.rsyslog.com/professional-services/
>> >> >> > What's up with rsyslog? Follow https://twitter.com/rgerhards
>> >> >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> >> >> > myriad
>> >> >> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
>> >> >> > you
>> >> >> > DON'T LIKE THAT.
>> >> >> _______________________________________________
>> >> >> rsyslog mailing list
>> >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >> >> http://www.rsyslog.com/professional-services/
>> >> >> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> >> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> >> >> myriad
>> >> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
>> you
>> >> >> DON'T LIKE THAT.
>> >> >>
>> >> > _______________________________________________
>> >> > rsyslog mailing list
>> >> > http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >> > http://www.rsyslog.com/professional-services/
>> >> > What's up with rsyslog? Follow https://twitter.com/rgerhards
>> >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> myriad
>> >> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
>> you DON'T
>> >> > LIKE THAT.
>> >>
>> >>
>> >>
>> >> --
>> >> Regards,
>> >> Janmejay
>> >> http://codehunk.wordpress.com
>>
>>
>>
>> --
>> Regards,
>> Janmejay
>> http://codehunk.wordpress.com
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
--
Regards,
Janmejay
http://codehunk.wordpress.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
