Can you reproduce the failure with synchronously calling mmjsonparse?
I'll try to reproduce it again.
--
Regards,
Janmejay
PS: Please blame the typos in this mail on my phone's uncivilized soft
keyboard sporting it's not-so-smart-assist technology.
On Apr 9, 2015 3:20 PM, "chenlin rao" <rao.chen...@gmail.com> wrote:
> janmyjay:
> I don't think that's a json-c problem, because if I use omfile inside
> foreach, rsyslogd never died.
> I also try to compile and run your repo, process fault again and the
> valgrind output keep totally same.
>
> 2015-04-08 21:26 GMT+08:00 singh.janmejay <singh.janme...@gmail.com>:
>
> > BTW, here is the gdb-session-trace if you are interested:
> >
> > Program received signal SIGSEGV, Segmentation fault.
> > [Switching to Thread 0x7ffff1657700 (LWP 14762)]
> > 0x00007ffff718c23b in array_list_get_idx () from
> > /lib/x86_64-linux-gnu/libjson-c.so.2
> > (gdb) bt
> > #0 0x00007ffff718c23b in array_list_get_idx () from
> > /lib/x86_64-linux-gnu/libjson-c.so.2
> > #1 0x00000000004450af in execForeach (stmt=0x6ffab0,
> > pMsg=0x7fffec016040, pWti=0x718810) at ruleset.c:279
> > #2 0x0000000000445874 in scriptExec (root=0x6ffab0,
> > pMsg=0x7fffec016040, pWti=0x718810) at ruleset.c:464
> > #3 0x0000000000444fdb in execIf (stmt=0x6ffdc0, pMsg=0x7fffec016040,
> > pWti=0x718810) at ruleset.c:257
> > #4 0x000000000044584a in scriptExec (root=0x6fd560,
> > pMsg=0x7fffec016040, pWti=0x718810) at ruleset.c:461
> > #5 0x0000000000444ebc in execCall (stmt=0x6fd4c0,
> > pMsg=0x7fffec016040, pWti=0x718810) at ruleset.c:232
> > #6 0x0000000000445820 in scriptExec (root=0x6fd4c0,
> > pMsg=0x7fffec016040, pWti=0x718810) at ruleset.c:458
> > #7 0x00000000004459f2 in processBatch (pBatch=0x718848,
> > pWti=0x718810) at ruleset.c:503
> > #8 0x000000000045b390 in msgConsumer (notNeeded=0x0, pBatch=0x718848,
> > pWti=0x718810) at rsyslogd.c:575
> > #9 0x00000000004409f3 in ConsumerReg (pThis=0x718380, pWti=0x718810)
> > at queue.c:1897
> > #10 0x000000000043bcfd in wtiWorker (pThis=0x718810) at wti.c:334
> > #11 0x000000000043a8af in wtpWorker (arg=0x718810) at wtp.c:389
> > #12 0x00007ffff79ac0a4 in start_thread (arg=0x7ffff1657700) at
> > pthread_create.c:309
> > #13 0x00007ffff6abc04d in clone () at
> > ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
> > (gdb) up 1
> > #1 0x00000000004450af in execForeach (stmt=0x6ffab0,
> > pMsg=0x7fffec016040, pWti=0x718810) at ruleset.c:279
> > 279 curr = json_object_array_get_idx(arr, i);
> > (gdb) p pMsg->json
> > $1 = (struct json_object *) 0x7fffe00176b0
> > (gdb) p *pMsg->json
> > $2 = {o_type = json_type_object, _delete = 0x7ffff718c9a0,
> > _to_json_string = 0x7ffff718cee0, _ref_count = 1, _pb = 0x0, o =
> > {c_boolean = -536774912,
> > c_double = 6.9533292876306475e-310, c_int64 = 140736951580416,
> > c_object = 0x7fffe0017700, c_array = 0x7fffe0017700, c_string = {str =
> > 0x7fffe0017700 "\020", len = 0}},
> > _user_delete = 0x0, _userdata = 0x0}
> > (gdb) p *pMsg->json->o.c_object
> > $3 = {size = 16, count = 1, collisions = 0, resizes = 0, lookups = 2,
> > inserts = 1, deletes = 0, name = 0x0, head = 0x7fffe00177a0, tail =
> > 0x7fffe00177a0, table = 0x7fffe0017760,
> > free_fn = 0x7ffff718cca0, hash_fn = 0x7ffff718fd60 <lh_char_hash>,
> > equal_fn = 0x7ffff71901f0 <lh_char_equal>}
> > (gdb) p *pMsg->json->o.c_object->head->key
> > There is no member named key.
> > (gdb) p pMsg->json->o.c_object->head->key
> > There is no member named key.
> > (gdb) p *pMsg->json->o.c_object->head
> > $4 = {k = 0x7fffe0018230, v = 0x7fffe0017970, next = 0x0, prev = 0x0}
> > (gdb) p *pMsg->json->o.c_object->head->k
> > Attempt to dereference a generic pointer.
> > (gdb) p (char*) pMsg->json->o.c_object->head->k
> > $5 = 0x7fffe0018230 "msg"
> > (gdb) p (json_object*) pMsg->json->o.c_object->head->v
> > $6 = (json_object *) 0x7fffe0017970
> > (gdb) p *(json_object*) pMsg->json->o.c_object->head->v
> > $7 = {o_type = json_type_string, _delete = 0x7ffff718c9c0,
> > _to_json_string = 0x7ffff718c930, _ref_count = 1, _pb = 0x0, o =
> > {c_boolean = -536774208,
> > c_double = 6.9533292876654297e-310, c_int64 = 140736951581120,
> > c_object = 0x7fffe00179c0, c_array = 0x7fffe00179c0, c_string = {
> > str = 0x7fffe00179c0 "[{\"content\":\"Error
> > Domain=WBBussinessErrorDomain Code=20030 \\\"the user base info is not
> > exists!\\\" UserInfo=0x17127cc00 {NSLocalizedDescription=the user base
> > info is not exists!, WBBussinessErrorInfoKe"..., len = 2144}},
> > _user_delete = 0x0, _userdata = 0x0}
> >
> > On Wed, Apr 8, 2015 at 6:53 PM, singh.janmejay <singh.janme...@gmail.com
> >
> > wrote:
> > > Spent some time on this today.
> > > I had the crash reproducing consistently with tcpflood -C 1000 ...
> > >
> > > Here is what I observed:
> > > - When large number of messages (copy of the message you shared) are
> > > sent, some messages get parsed with key 'msg' pointing to a
> > > json-string (its contents is the entire remaining string, starting
> > > with [{"content....
> > > - I have added a type-check which ensures foreach tries to iterate
> > > only when the given type is an array, if not, it prints out a warning
> > > and skips the loop.
> > >
> > > This means, it'll skip ill-parsed messages. _I haven't looked through
> > > the json-c parser code to reason code-flow that can lead to this
> > > problem_, but because of it happening on a different thread, changes
> > > may be visible to ruleset thread in unexpected order, which would make
> > > it see the json-object being available, but its state being
> > > intermediate (rather than final).
> > >
> > > Can you test with git://github.com/janmejay/rsyslog.git and let us
> > > know if it fixes the crash?
> > >
> > > With this change I can't reproduce the crash anymore, but again,
> > > because modifications to the json-object are happening on a different
> > > thread, it could lead to crash on accessing that object for a
> > > different reason. However, this also means the ill-parsed messages
> > > will be skipped, so depending on how you reason about it, they will be
> > > lost.
> > >
> > > While this fixes the value not being an array problem, you still want
> > > to move to sync mmjsonparse call.
> > >
> > >
> > >
> > > On Mon, Apr 6, 2015 at 6:07 PM, chenlin rao <rao.chen...@gmail.com>
> > wrote:
> > >> yes.
> > >> rsyslogd 8.8.0.ad1, compiled with:
> > >> PLATFORM: x86_64-redhat-linux-gnu
> > >> PLATFORM (lsb_release -d):
> > >> FEATURE_REGEXP: Yes
> > >> GSSAPI Kerberos 5 support: No
> > >> FEATURE_DEBUG (debug build, slow code): No
> > >> 32bit Atomic operations supported: Yes
> > >> 64bit Atomic operations supported: Yes
> > >> memory allocator: system default
> > >> Runtime Instrumentation (slow code): No
> > >> uuid support: Yes
> > >> Number of Bits in RainerScript integers: 64
> > >>
> > >> 2015-04-06 20:32 GMT+08:00 singh.janmejay <singh.janme...@gmail.com>:
> > >>
> > >>> So it is 8.8.0?
> > >>>
> > >>> On Mon, Apr 6, 2015 at 5:33 PM, chenlin rao <rao.chen...@gmail.com>
> > wrote:
> > >>> > I install latest version from rsyslog_v8 yum repo in CentOS6.5.
> > >>> > And for more options in valgrind report, I compiled another version
> > from
> > >>> > git master, ran into the same problem.
> > >>> >
> > >>> > 2015-04-06 13:55 GMT+08:00 singh.janmejay <
> singh.janme...@gmail.com
> > >:
> > >>> >
> > >>> >> It was looking very similar to the issue that Thomas reported on
> > >>> >>
> > >>> >>
> > >>>
> >
> http://www.gossamer-threads.com/lists/rsyslog/users/16204?do=post_view_threaded
> > >>> >>
> > >>> >> To double-check, I cherry-picked the two patches in that PR, which
> > >>> >> fixed the failing json_array_looping test.
> > >>> >>
> > >>> >> So I'll now focus on Chenlin's problem.
> > >>> >>
> > >>> >> To rule-out the same problem (the one that PR fixes), Chenlin, can
> > you
> > >>> >> confirm which version of rsyslog are you on?
> > >>> >>
> > >>> >>
> > >>> >>
> > >>> >> On Thu, Apr 2, 2015 at 6:58 PM, singh.janmejay <
> > >>> singh.janme...@gmail.com>
> > >>> >> wrote:
> > >>> >> > I couldn't spend too much time on it, but just saw it reproduce
> > on my
> > >>> >> > local env (also running Ubuntu 12.04).
> > >>> >> >
> > >>> >> > Will again get a chance to look at in a a few days (likely
> > Monday).
> > >>> >> >
> > >>> >> > Again, sorry for the delay.
> > >>> >> >
> > >>> >> > On Thu, Apr 2, 2015 at 6:43 PM, singh.janmejay <
> > >>> singh.janme...@gmail.com>
> > >>> >> wrote:
> > >>> >> >> Foreach support is a recent addition.
> > >>> >> >>
> > >>> >> >> On Thu, Apr 2, 2015 at 3:34 PM, singh.janmejay <
> > >>> >> singh.janme...@gmail.com> wrote:
> > >>> >> >>> I saw that mail, but by that time I had already setup the
> > >>> environment
> > >>> >> >>> to reproduce it with Chenlin's config.
> > >>> >> >>>
> > >>> >> >>> I should pursue the CI failure first. Didn't know it was
> > happening
> > >>> in
> > >>> >> every run.
> > >>> >> >>>
> > >>> >> >>> On Thu, Apr 2, 2015 at 2:04 PM, Rainer Gerhards
> > >>> >> >>> <rgerha...@hq.adiscon.com> wrote:
> > >>> >> >>>> I
> > >>> >> >>>>
> > >>> >> >>>> 2015-04-02 7:35 GMT+02:00 singh.janmejay <
> > singh.janme...@gmail.com
> > >>> >:
> > >>> >> >>>>> Hadn't noticed the use of msg key early-on, so "]}"
> shouldn't
> > be a
> > >>> >> >>>>> problem, but I can't reproduce it with input i crafted to
> work
> > >>> with
> > >>> >> >>>>> it.
> > >>> >> >>>>> Here is the crafted input:
> > >>> >> >>>>> @cee:{"msg" : ["G\"foo\":\"10\"}", "G\"bar\":\"20\""]}
> > >>> >> >>>>>
> > >>> >> >>>>> This is what does fail it, im using a bad type (it doesn't
> do
> > >>> >> type-check yet):
> > >>> >> >>>>> @cee:{"msg" : "G\"foo\":\"10\"}", "bar" : ["foo"]}
> > >>> >> >>>>>
> > >>> >> >>>>> Here it ending in value of 'bar' satisfies the ']}'
> > conditional,
> > >>> but
> > >>> >> >>>>> msg isn't an array. I'll add array-typecheck, but other than
> > this
> > >>> I
> > >>> >> >>>>> have been unable to reproduce it, so this may not fix your
> > >>> problem.
> > >>> >> >>>>>
> > >>> >> >>>>> I think your input line (or entire input file) will help.
> > >>> >> >>>>
> > >>> >> >>>> No sure if you overlooked that posting. As it looks, the
> > problem
> > >>> seems
> > >>> >> >>>> to be reproducible with the testbench as well. At least this
> > commit
> > >>> >> >>>> continously fails:
> > >>> >> >>>>
> > >>> >> >>>> https://github.com/rsyslog/rsyslog/pull/286
> > >>> >> >>>>
> > >>> >> >>>> I've also sent a build log via te ML a couple of days ago.
> > >>> >> >>>>
> > >>> >> >>>> HTH
> > >>> >> >>>> Rainer
> > >>> >> >>>>>
> > >>> >> >>>>> On Thu, Apr 2, 2015 at 9:39 AM, singh.janmejay <
> > >>> >> singh.janme...@gmail.com> wrote:
> > >>> >> >>>>>> Hi Chenlin,
> > >>> >> >>>>>>
> > >>> >> >>>>>> I finally have some time to look at it. (Sorry for the
> > delay.)
> > >>> >> >>>>>>
> > >>> >> >>>>>> Can you please share a sample log-line? Feel free to
> > anonymize
> > >>> it,
> > >>> >> but
> > >>> >> >>>>>> please keep the structure and types as is.
> > >>> >> >>>>>>
> > >>> >> >>>>>> I see that you are checking $msg ends with "]}", that means
> > it
> > >>> is an
> > >>> >> >>>>>> object, not an array.
> > >>> >> >>>>>>
> > >>> >> >>>>>> If my interpretation is correct, trying to iterate over
> > object
> > >>> >> instead
> > >>> >> >>>>>> of array may be causing this problem. (I haven't looked at
> > code
> > >>> yet
> > >>> >> to
> > >>> >> >>>>>> verify this manifestation though).
> > >>> >> >>>>>>
> > >>> >> >>>>>> On Tue, Mar 31, 2015 at 2:32 PM, Rainer Gerhards
> > >>> >> >>>>>> <rgerha...@hq.adiscon.com> wrote:
> > >>> >> >>>>>>> 2015-03-30 19:55 GMT+02:00 singh.janmejay <
> > >>> >> singh.janme...@gmail.com>:
> > >>> >> >>>>>>>> Sorry, I haven't had a chance to look at it yet.
> > >>> >> >>>>>>>
> > >>> >> >>>>>>> Take your time, I know how challenging it is at times ;) I
> > just
> > >>> >> wanted
> > >>> >> >>>>>>> to spread the good news that we have a good repro.
> > >>> >> >>>>>>>
> > >>> >> >>>>>>> Rainer
> > >>> >> >>>>>>>>
> > >>> >> >>>>>>>> Will get to this asap.
> > >>> >> >>>>>>>>
> > >>> >> >>>>>>>> On Mon, Mar 30, 2015 at 2:36 PM, Rainer Gerhards
> > >>> >> >>>>>>>> <rgerha...@hq.adiscon.com> wrote:
> > >>> >> >>>>>>>>> Co-incidentally, an unrelated testbench run also
> > experienced
> > >>> >> problems
> > >>> >> >>>>>>>>> in this area. I have attached the testbench log.
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> Here is the valgrind report:
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> 5812.499032607:main Q:Reg/w0 : wti 0x614e9e0: we need
> to
> > >>> create
> > >>> >> a new
> > >>> >> >>>>>>>>> action worker instance for action 2
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> 5812.499181273:main Q:Reg/w0 : Action 2 transitioned to
> > >>> state:
> > >>> >> itx
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> 5812.499314576:main Q:Reg/w0 : FOREACH .corge IN
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> 5812.499625664:main Q:Reg/w0 : var '.quux!bar'
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> 5812.500078770:main Q:Reg/w0 : eval expr 0x60849e0,
> type
> > >>> 'V[86]'
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== Thread 6:
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== Invalid read of size 8
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== at 0x587B134: lh_table_lookup_entry (in
> > >>> >> >>>>>>>>> /usr/lib/x86_64-linux-gnu/libjson.so.0.0.1)
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== by 0x587B1A8: lh_table_lookup (in
> > >>> >> >>>>>>>>> /usr/lib/x86_64-linux-gnu/libjson.so.0.0.1)
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== by 0x416F98: jsonVarExtract (msg.c:4218)
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== by 0x41C82F: msgGetJSONPropJSON (msg.c:2848)
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== by 0x4562D0: cnfexprEval
> (rainerscript.c:1785)
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== by 0x45842E: cnfexprEvalCollection
> > >>> >> (rainerscript.c:2483)
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== by 0x435CFC: scriptExec (ruleset.c:271)
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== by 0x435D6B: scriptExec (ruleset.c:284)
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== by 0x436233: processBatch (ruleset.c:503)
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== by 0x44464B: msgConsumer (rsyslogd.c:576)
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== by 0x432AC2: ConsumerReg (queue.c:1897)
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== by 0x42F55E: wtiWorker (wti.c:334)
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== Address 0x61645c8 is 8 bytes before a block of
> > size
> > >>> 5
> > >>> >> alloc'd
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== at 0x4C2B6CD: malloc (in
> > >>> >> >>>>>>>>> /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== by 0x5D0B081: strdup (strdup.c:43)
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== by 0x5878A5C: json_object_object_add (in
> > >>> >> >>>>>>>>> /usr/lib/x86_64-linux-gnu/libjson.so.0.0.1)
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== by 0x41E419: msgAddJSON (msg.c:4379)
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== by 0x41F2E0: msgSetJSONFromVar (msg.c:4554)
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== by 0x435D4E: scriptExec (ruleset.c:283)
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== by 0x436233: processBatch (ruleset.c:503)
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== by 0x44464B: msgConsumer (rsyslogd.c:576)
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== by 0x432AC2: ConsumerReg (queue.c:1897)
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== by 0x42F55E: wtiWorker (wti.c:334)
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== by 0x42E6C6: wtpWorker (wtp.c:389)
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== by 0x5050E99: start_thread
> > (pthread_create.c:308)
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284==
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== Jump to the invalid address stated on the next
> > line
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== at 0x0: ???
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== by 0x587B136: lh_table_lookup_entry (in
> > >>> >> >>>>>>>>> /usr/lib/x86_64-linux-gnu/libjson.so.0.0.1)
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== by 0x587B1A8: lh_table_lookup (in
> > >>> >> >>>>>>>>> /usr/lib/x86_64-linux-gnu/libjson.so.0.0.1)
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== by 0x416F98: jsonVarExtract (msg.c:4218)
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== by 0x41C82F: msgGetJSONPropJSON (msg.c:2848)
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== by 0x4562D0: cnfexprEval
> (rainerscript.c:1785)
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== by 0x45842E: cnfexprEvalCollection
> > >>> >> (rainerscript.c:2483)
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== by 0x435CFC: scriptExec (ruleset.c:271)
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== by 0x435D6B: scriptExec (ruleset.c:284)
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== by 0x436233: processBatch (ruleset.c:503)
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== by 0x44464B: msgConsumer (rsyslogd.c:576)
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== by 0x432AC2: ConsumerReg (queue.c:1897)
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== Address 0x0 is not stack'd, malloc'd or
> > (recently)
> > >>> >> free'd
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284==
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284==
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== Process terminating with default action of
> > signal 11
> > >>> >> (SIGSEGV)
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== Bad permissions for mapped region at address
> 0x0
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== at 0x0: ???
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== by 0x587B136: lh_table_lookup_entry (in
> > >>> >> >>>>>>>>> /usr/lib/x86_64-linux-gnu/libjson.so.0.0.1)
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== by 0x587B1A8: lh_table_lookup (in
> > >>> >> >>>>>>>>> /usr/lib/x86_64-linux-gnu/libjson.so.0.0.1)
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== by 0x416F98: jsonVarExtract (msg.c:4218)
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== by 0x41C82F: msgGetJSONPropJSON (msg.c:2848)
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== by 0x4562D0: cnfexprEval
> (rainerscript.c:1785)
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== by 0x45842E: cnfexprEvalCollection
> > >>> >> (rainerscript.c:2483)
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== by 0x435CFC: scriptExec (ruleset.c:271)
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== by 0x435D6B: scriptExec (ruleset.c:284)
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== by 0x436233: processBatch (ruleset.c:503)
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== by 0x44464B: msgConsumer (rsyslogd.c:576)
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== by 0x432AC2: ConsumerReg (queue.c:1897)
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284==
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== HEAP SUMMARY:
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== in use at exit: 961,978 bytes in 1,632
> blocks
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== total heap usage: 2,448 allocs, 816 frees,
> > >>> 1,023,184
> > >>> >> bytes allocated
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284==
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== LEAK SUMMARY:
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== definitely lost: 6 bytes in 1 blocks
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== indirectly lost: 0 bytes in 0 blocks
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== possibly lost: 2,592 bytes in 9 blocks
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== still reachable: 959,380 bytes in 1,622
> blocks
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== suppressed: 0 bytes in 0 blocks
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== Rerun with --leak-check=full to see details of
> > leaked
> > >>> >> memory
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284==
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== For counts of detected and suppressed errors,
> > rerun
> > >>> >> with: -v
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ==2284== ERROR SUMMARY: 2 errors from 2 contexts
> > (suppressed:
> > >>> 2
> > >>> >> from 2)
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> receiving response: Connection reset by peer
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> ./diag.sh: line 17: 2284 Killed
> > $valgrind
> > >>> >> >>>>>>>>> ../tools/rsyslogd -C -n -irsyslog$3.pid
> > >>> >> -M../runtime/.libs:../.libs
> > >>> >> >>>>>>>>> -f$srcdir/testsuites/$2
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> FAIL: json_array_looping.sh
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> Rainer
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> 2015-03-21 17:50 GMT+01:00 chenlin rao <
> > rao.chen...@gmail.com
> > >>> >:
> > >>> >> >>>>>>>>>> thanks for the information. But rsyslogd also fault
> > after I
> > >>> >> change
> > >>> >> >>>>>>>>>> mmjsonparse action config as `action (
> type="mmjsonparse"
> > >>> >> >>>>>>>>>> name="action_jsonarray-parse"`. output as follow:
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> 6008.997308131:main Q:Reg/w1 : eval expr 0x52f18f0,
> > return
> > >>> >> datatype 'J'
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> 6009.002050327:main Q:Reg/w1 : ACTION 16
> > >>> >> >>>>>>>>>> [builtin:omfwd:action(type="builtin:omfwd" ...)]
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> 6009.002357554:main Q:Reg/w1 : executing action 16
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> 6009.002454575:main Q:Reg/w1 : Called action, logging
> to
> > >>> >> builtin:omfwd
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> ==9947== Thread 22:
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> ==9947== Invalid read of size 4
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> ==9947== at 0x3A044093A0: pthread_mutex_lock (in
> > /lib64/
> > >>> >> >>>>>>>>>> libpthread-2.12.so)
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> ==9947== by 0x438053: qqueueEnqMsg (queue.c:2856)
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> ==9947== by 0x4421AB: doSubmitToActionQ
> > (action.c:1403)
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> ==9947== by 0x439D6C: scriptExec (ruleset.c:197)
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> ==9947== by 0x439ECC: scriptExec (ruleset.c:284)
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> ==9947== by 0x439CD9: scriptExec (ruleset.c:416)
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> ==9947== by 0x439CD9: scriptExec (ruleset.c:416)
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> ==9947== by 0x439CD9: scriptExec (ruleset.c:416)
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> ==9947== by 0x43A40B: processBatch (ruleset.c:503)
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> ==9947== by 0x44A403: msgConsumer (rsyslogd.c:575)
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> ==9947== by 0x438D12: ConsumerReg (queue.c:1897)
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> ==9947== by 0x43314B: wtiWorker (wti.c:334)
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> ==9947== Address 0x10 is not stack'd, malloc'd or
> > (recently)
> > >>> >> free'd
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> ==9947==
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> ==9947==
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> ==9947== Process terminating with default action of
> > signal 11
> > >>> >> (SIGSEGV)
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> ==9947== Access not within mapped region at address
> 0x10
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> ==9947== at 0x3A044093A0: pthread_mutex_lock (in
> > /lib64/
> > >>> >> >>>>>>>>>> libpthread-2.12.so)
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> ==9947== by 0x438053: qqueueEnqMsg (queue.c:2856)
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> ==9947== by 0x4421AB: doSubmitToActionQ
> > (action.c:1403)
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> ==9947== by 0x439D6C: scriptExec (ruleset.c:197)
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> ==9947== by 0x439ECC: scriptExec (ruleset.c:284)
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> ==9947== by 0x439CD9: scriptExec (ruleset.c:416)
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> ==9947== by 0x439CD9: scriptExec (ruleset.c:416)
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> ==9947== by 0x439CD9: scriptExec (ruleset.c:416)
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> ==9947== by 0x43A40B: processBatch (ruleset.c:503)
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> ==9947== by 0x44A403: msgConsumer (rsyslogd.c:575)
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> ==9947== by 0x438D12: ConsumerReg (queue.c:1897)
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> ==9947== by 0x43314B: wtiWorker (wti.c:334)
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> ==9947== If you believe this happened as a result of a
> > stack
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> ==9947== overflow in your program's main thread
> > (unlikely
> > >>> but
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> ==9947== possible), you can try to increase the size
> of
> > the
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> ==9947== main thread stack using the --main-stacksize=
> > flag.
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> ==9947== The main thread stack size used in this run
> was
> > >>> >> 10485760.
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> ==9947==
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> ==9947== HEAP SUMMARY:
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> ==9947== in use at exit: 1,181,215,468 bytes in
> > 282,120
> > >>> >> blocks
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> ==9947== total heap usage: 379,556 allocs, 97,436
> > frees,
> > >>> >> 2,096,403,341
> > >>> >> >>>>>>>>>> bytes allocated
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> ==9947==
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> ==9947== LEAK SUMMARY:
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> ==9947== definitely lost: 516 bytes in 12 blocks
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> ==9947== indirectly lost: 0 bytes in 0 blocks
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> ==9947== possibly lost: 33,561,793 bytes in 24
> > blocks
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> ==9947== still reachable: 1,147,653,159 bytes in
> > 282,084
> > >>> >> blocks
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> ==9947== suppressed: 0 bytes in 0 blocks
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> ==9947== Rerun with --leak-check=full to see details of
> > >>> leaked
> > >>> >> memory
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> ==9947==
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> ==9947== For counts of detected and suppressed errors,
> > rerun
> > >>> >> with: -v
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> ==9947== Use --track-origins=yes to see where
> > uninitialised
> > >>> >> values come from
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> ==9947== ERROR SUMMARY: 25 errors from 5 contexts
> > >>> (suppressed:
> > >>> >> 100 from 9)
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> Seems very similar with before.
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> BTW: is it the "workerthreads" options make sense to
> > >>> >> mmjsonparse/mmfields
> > >>> >> >>>>>>>>>> etc?
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>> 2015-03-22 0:07 GMT+08:00 Rainer Gerhards <
> > >>> >> rgerha...@hq.adiscon.com>:
> > >>> >> >>>>>>>>>>
> > >>> >> >>>>>>>>>>> 2015-03-21 14:50 GMT+01:00 chenlin rao <
> > >>> rao.chen...@gmail.com
> > >>> >> >:
> > >>> >> >>>>>>>>>>> > $MaxMessageSize 32m
> > >>> >> >>>>>>>>>>> > module( load="imtcp" )
> > >>> >> >>>>>>>>>>> > module( load="imuxsock" )
> > >>> >> >>>>>>>>>>> > module( load="imklog" )
> > >>> >> >>>>>>>>>>> > module( load="mmfields" )
> > >>> >> >>>>>>>>>>> > module( load="mmjsonparse" )
> > >>> >> >>>>>>>>>>> > module( load="omelasticsearch" )
> > >>> >> >>>>>>>>>>> > $ActionFileDefaultTemplate
> > RSYSLOG_TraditionalFileFormat
> > >>> >> >>>>>>>>>>> > $RepeatedMsgReduction off
> > >>> >> >>>>>>>>>>> > $WorkDirectory /data0/rsyslog
> > >>> >> >>>>>>>>>>> > $MainMsgQueueFilename mainQ
> > >>> >> >>>>>>>>>>> > $MainMsgQueueType LinkedList
> > >>> >> >>>>>>>>>>> > $MainMsgQueueMaxFileSize 512M
> > >>> >> >>>>>>>>>>> > $MainMsgQueueMaxDiskSpace 20G
> > >>> >> >>>>>>>>>>> > $PreserveFQDN on
> > >>> >> >>>>>>>>>>> > input(type="imtcp" port="514")
> > >>> >> >>>>>>>>>>> > template(name="local6IndexName" type="list") {
> > >>> >> >>>>>>>>>>> > constant(value="logstash-mweibo-")
> > >>> >> >>>>>>>>>>> > property(name="timereported"
> dateFormat="rfc3339"
> > >>> >> position.from="1"
> > >>> >> >>>>>>>>>>> > position.to="4")
> > >>> >> >>>>>>>>>>> > constant(value=".")
> > >>> >> >>>>>>>>>>> > property(name="timereported"
> dateFormat="rfc3339"
> > >>> >> position.from="6"
> > >>> >> >>>>>>>>>>> > position.to="7")
> > >>> >> >>>>>>>>>>> > constant(value=".")
> > >>> >> >>>>>>>>>>> > property(name="timereported"
> dateFormat="rfc3339"
> > >>> >> position.from="9"
> > >>> >> >>>>>>>>>>> > position.to="10")
> > >>> >> >>>>>>>>>>> > }
> > >>> >> >>>>>>>>>>> > template( name="local6TypeName" type="string"
> > >>> >> string="%programname%" )
> > >>> >> >>>>>>>>>>> > template( name="local6JsonArray" type="list" ) {
> > >>> >> >>>>>>>>>>> > constant(value="{\"@timestamp\":\"")
> > >>> >> property(name="timereported"
> > >>> >> >>>>>>>>>>> > dateFormat="rfc3339")
> > >>> >> >>>>>>>>>>> > constant(value="\",\"host\":\"")
> > >>> >> property(name="hostname")
> > >>> >> >>>>>>>>>>> > constant(value="\",") property(name="$.line"
> > >>> >> position.from="2")
> > >>> >> >>>>>>>>>>> > }
> > >>> >> >>>>>>>>>>> > Ruleset( name="forwardRuleSetJsonArray" )
> > >>> >> >>>>>>>>>>> > {
> > >>> >> >>>>>>>>>>> > action ( type="mmjsonparse"
> > >>> >> >>>>>>>>>>> > name="action_jsonarray-parse"
> > >>> >> >>>>>>>>>>> > queue.size="3000"
> > >>> >> >>>>>>>>>>> > queue.dequeuebatchsize="300"
> > >>> >> >>>>>>>>>>> > queue.maxdiskspace="15G"
> > >>> >> >>>>>>>>>>> > queue.checkpointinterval="10"
> > >>> >> >>>>>>>>>>> > queue.type="linkedlist"
> > >>> >> >>>>>>>>>>> > queue.workerthreads="30"
> > >>> >> >>>>>>>>>>> > queue.workerthreadminimummessages="100"
> > >>> >> >>>>>>>>>>> > queue.maxfilesize="500M"
> > >>> >> >>>>>>>>>>> > queue.saveonshutdown="on"
> > >>> >> >>>>>>>>>>> > )
> > >>> >> >>>>>>>>>>>
> > >>> >> >>>>>>>>>>> running mmjsonparse on a queue (asynchronously!) does
> > not
> > >>> make
> > >>> >> any
> > >>> >> >>>>>>>>>>> sense. When running async, the json properties will
> > never be
> > >>> >> seen by
> > >>> >> >>>>>>>>>>> the rest of the ruleset.
> > >>> >> >>>>>>>>>>>
> > >>> >> >>>>>>>>>>> Maybe this even triggers the fault...
> > >>> >> >>>>>>>>>>>
> > >>> >> >>>>>>>>>>> Rainer
> > >>> >> >>>>>>>>>>> > # maybe recieved a too large data being truncated
> > >>> >> >>>>>>>>>>> > if ( re_match($msg, ']}$') ) then {
> > >>> >> >>>>>>>>>>> > foreach ($.line in $!msg) do {
> > >>> >> >>>>>>>>>>> > action (
> > >>> >> >>>>>>>>>>> > # test for other output modules
> > >>> >> >>>>>>>>>>> > # type="omfile"
> > >>> file="/data/rsyslog-debug.log"
> > >>> >> >>>>>>>>>>> > # type="omfwd" Target="127.0.0.1"
> > >>> Port="5140"
> > >>> >> >>>>>>>>>>> Protocol="tcp"
> > >>> >> >>>>>>>>>>> > type="omelasticsearch" server="
> > >>> es.domain.com
> > >>> >> "
> > >>> >> >>>>>>>>>>> > dynSearchIndex="on" searchIndex="local6IndexName"
> > >>> >> dynSearchType="on"
> > >>> >> >>>>>>>>>>> > searchType="local6TypeName" asyncrepl="on"
> > bulkmode="on"
> > >>> >> >>>>>>>>>>> > template="local6JsonArray"
> > >>> >> >>>>>>>>>>> > name="action_json2log-es1003"
> > >>> >> >>>>>>>>>>> >
> > queue.filename="action_json2log-es1003"
> > >>> >> >>>>>>>>>>> > queue.size="10000"
> > >>> >> >>>>>>>>>>> > queue.dequeuebatchsize="2000"
> > >>> >> >>>>>>>>>>> > queue.maxdiskspace="15G"
> > >>> >> >>>>>>>>>>> > queue.discardseverity="3"
> > >>> >> >>>>>>>>>>> > queue.checkpointinterval="10"
> > >>> >> >>>>>>>>>>> > queue.type="linkedlist"
> > >>> >> >>>>>>>>>>> > queue.workerthreads="5"
> > >>> >> >>>>>>>>>>> >
> > queue.workerthreadminimummessages="2000"
> > >>> >> >>>>>>>>>>> > queue.maxfilesize="500M"
> > >>> >> >>>>>>>>>>> > queue.saveonshutdown="on"
> > >>> >> >>>>>>>>>>> > )
> > >>> >> >>>>>>>>>>> > }
> > >>> >> >>>>>>>>>>> > stop
> > >>> >> >>>>>>>>>>> > }
> > >>> >> >>>>>>>>>>> > }
> > >>> >> >>>>>>>>>>> >
> > >>> >> >>>>>>>>>>> > if ( $programname ==
> 'mobile_client_net_fatal_error' )
> > >>> then
> > >>> >> >>>>>>>>>>> > {
> > >>> >> >>>>>>>>>>> > call forwardRuleSetJsonArray
> > >>> >> >>>>>>>>>>> > stop
> > >>> >> >>>>>>>>>>> > }
> > >>> >> >>>>>>>>>>> >
> > >>> >> >>>>>>>>>>>
> > >>> >>
> > >>>
> >
> *.info;mail.none;authpriv.none;cron.none;local6.none;local7.none;user.none
> > >>> >> >>>>>>>>>>> > /var/log/messages
> > >>> >> >>>>>>>>>>> > authpriv.*
> > >>> >> /var/log/secure
> > >>> >> >>>>>>>>>>> > mail.*
> > >>> >> /var/log/maillog
> > >>> >> >>>>>>>>>>> > cron.*
> > >>> >> /var/log/cron
> > >>> >> >>>>>>>>>>> > uucp,news.crit
> > >>> >> /var/log/spooler
> > >>> >> >>>>>>>>>>> >
> > >>> >> >>>>>>>>>>> > 2015-03-21 13:14 GMT+08:00 singh.janmejay <
> > >>> >> singh.janme...@gmail.com>:
> > >>> >> >>>>>>>>>>> >
> > >>> >> >>>>>>>>>>> >> Can you please share your config as well?
> > >>> >> >>>>>>>>>>> >>
> > >>> >> >>>>>>>>>>> >> Also, I'll likely be able to look at it only after
> > 27th
> > >>> >> Mar. As of now I
> > >>> >> >>>>>>>>>>> >> don't have access to a computer and github doesn't
> > seem
> > >>> to
> > >>> >> show line
> > >>> >> >>>>>>>>>>> >> numbers while browsing code using mobile phone.
> > >>> >> >>>>>>>>>>> >>
> > >>> >> >>>>>>>>>>> >> --
> > >>> >> >>>>>>>>>>> >> Regards,
> > >>> >> >>>>>>>>>>> >> Janmejay
> > >>> >> >>>>>>>>>>> >>
> > >>> >> >>>>>>>>>>> >> PS: Please blame the typos in this mail on my
> phone's
> > >>> >> uncivilized soft
> > >>> >> >>>>>>>>>>> >> keyboard sporting it's not-so-smart-assist
> > technology.
> > >>> >> >>>>>>>>>>> >>
> > >>> >> >>>>>>>>>>> >> On Mar 21, 2015 8:49 AM, "chenlin rao" <
> > >>> >> rao.chen...@gmail.com> wrote:
> > >>> >> >>>>>>>>>>> >>
> > >>> >> >>>>>>>>>>> >> > I try to build rsyslogd from github master with
> > >>> >> "./configure
> > >>> >> >>>>>>>>>>> >> --enable-debug
> > >>> >> >>>>>>>>>>> >> > --enable-valgrind --enable-memcheck
> > >>> --enable-elasticsearch
> > >>> >> >>>>>>>>>>> >> > --enable-mmjsonparse --enable-mmsequence
> > >>> --enable-mmfields
> > >>> >> >>>>>>>>>>> >> > --disable-liblogging-stdlog --enable-omruleset"
> > >>> >> >>>>>>>>>>> >> >
> > >>> >> >>>>>>>>>>> >> > then process exit with:
> > >>> >> >>>>>>>>>>> >> >
> > >>> >> >>>>>>>>>>> >> > 7778.301430428:main Q[DA]:Reg/w0: ACTION 16
> > >>> >> >>>>>>>>>>> >> > [builtin:omfwd:action(type="builtin:omfwd" ...)]
> > >>> >> >>>>>>>>>>> >> > 7778.301676909:main Q[DA]:Reg/w0: executing
> action
> > 16
> > >>> >> >>>>>>>>>>> >> > 7778.301791807:main Q[DA]:Reg/w0: Called action,
> > >>> logging
> > >>> >> to
> > >>> >> >>>>>>>>>>> builtin:omfwd
> > >>> >> >>>>>>>>>>> >> > ==20833== Thread 2:
> > >>> >> >>>>>>>>>>> >> > ==20833== Invalid read of size 4
> > >>> >> >>>>>>>>>>> >> > ==20833== at 0x3A044093A0: pthread_mutex_lock
> > (in
> > >>> >> /lib64/
> > >>> >> >>>>>>>>>>> >> > libpthread-2.12.so)
> > >>> >> >>>>>>>>>>> >> > ==20833== by 0x438053: qqueueEnqMsg
> > (queue.c:2856)
> > >>> >> >>>>>>>>>>> >> > ==20833== by 0x4421AB: doSubmitToActionQ
> > >>> >> (action.c:1403)
> > >>> >> >>>>>>>>>>> >> > ==20833== by 0x439D6C: scriptExec
> > (ruleset.c:197)
> > >>> >> >>>>>>>>>>> >> > ==20833== by 0x439ECC: scriptExec
> > (ruleset.c:284)
> > >>> >> >>>>>>>>>>> >> > ==20833== by 0x439CD9: scriptExec
> > (ruleset.c:416)
> > >>> >> >>>>>>>>>>> >> > ==20833== by 0x43A40B: processBatch
> > (ruleset.c:503)
> > >>> >> >>>>>>>>>>> >> > ==20833== by 0x44A403: msgConsumer
> > (rsyslogd.c:575)
> > >>> >> >>>>>>>>>>> >> > ==20833== by 0x438D12: ConsumerReg
> > (queue.c:1897)
> > >>> >> >>>>>>>>>>> >> > ==20833== by 0x43314B: wtiWorker (wti.c:334)
> > >>> >> >>>>>>>>>>> >> > ==20833== by 0x431BA6: wtpWorker (wtp.c:389)
> > >>> >> >>>>>>>>>>> >> > ==20833== by 0x3A044079D0: start_thread (in
> > /lib64/
> > >>> >> >>>>>>>>>>> libpthread-2.12.so
> > >>> >> >>>>>>>>>>> >> )
> > >>> >> >>>>>>>>>>> >> > ==20833== Address 0x10 is not stack'd, malloc'd
> or
> > >>> >> (recently) free'd
> > >>> >> >>>>>>>>>>> >> > ==20833==
> > >>> >> >>>>>>>>>>> >> > ==20833==
> > >>> >> >>>>>>>>>>> >> > ==20833== Process terminating with default action
> > of
> > >>> >> signal 11
> > >>> >> >>>>>>>>>>> (SIGSEGV)
> > >>> >> >>>>>>>>>>> >> > ==20833== Access not within mapped region at
> > address
> > >>> 0x10
> > >>> >> >>>>>>>>>>> >> > ==20833== at 0x3A044093A0: pthread_mutex_lock
> > (in
> > >>> >> /lib64/
> > >>> >> >>>>>>>>>>> >> > libpthread-2.12.so)
> > >>> >> >>>>>>>>>>> >> > ==20833== by 0x438053: qqueueEnqMsg
> > (queue.c:2856)
> > >>> >> >>>>>>>>>>> >> > ==20833== by 0x4421AB: doSubmitToActionQ
> > >>> >> (action.c:1403)
> > >>> >> >>>>>>>>>>> >> > ==20833== by 0x439D6C: scriptExec
> > (ruleset.c:197)
> > >>> >> >>>>>>>>>>> >> > ==20833== by 0x439ECC: scriptExec
> > (ruleset.c:284)
> > >>> >> >>>>>>>>>>> >> > ==20833== by 0x439CD9: scriptExec
> > (ruleset.c:416)
> > >>> >> >>>>>>>>>>> >> > ==20833== by 0x43A40B: processBatch
> > (ruleset.c:503)
> > >>> >> >>>>>>>>>>> >> > ==20833== by 0x44A403: msgConsumer
> > (rsyslogd.c:575)
> > >>> >> >>>>>>>>>>> >> > ==20833== by 0x438D12: ConsumerReg
> > (queue.c:1897)
> > >>> >> >>>>>>>>>>> >> > ==20833== by 0x43314B: wtiWorker (wti.c:334)
> > >>> >> >>>>>>>>>>> >> > ==20833== by 0x431BA6: wtpWorker (wtp.c:389)
> > >>> >> >>>>>>>>>>> >> > ==20833== by 0x3A044079D0: start_thread (in
> > /lib64/
> > >>> >> >>>>>>>>>>> libpthread-2.12.so
> > >>> >> >>>>>>>>>>> >> )
> > >>> >> >>>>>>>>>>> >> > ==20833== If you believe this happened as a
> > result of
> > >>> a
> > >>> >> stack
> > >>> >> >>>>>>>>>>> >> > ==20833== overflow in your program's main thread
> > >>> >> (unlikely but
> > >>> >> >>>>>>>>>>> >> > ==20833== possible), you can try to increase the
> > size
> > >>> of
> > >>> >> the
> > >>> >> >>>>>>>>>>> >> > ==20833== main thread stack using the
> > >>> --main-stacksize=
> > >>> >> flag.
> > >>> >> >>>>>>>>>>> >> > ==20833== The main thread stack size used in
> this
> > run
> > >>> >> was 10485760.
> > >>> >> >>>>>>>>>>> >> > ==20833==
> > >>> >> >>>>>>>>>>> >> > ==20833== HEAP SUMMARY:
> > >>> >> >>>>>>>>>>> >> > ==20833== in use at exit: 44,605,326 bytes in
> > 5,019
> > >>> >> blocks
> > >>> >> >>>>>>>>>>> >> > ==20833== total heap usage: 10,310 allocs,
> 5,291
> > >>> frees,
> > >>> >> 78,621,963
> > >>> >> >>>>>>>>>>> >> bytes
> > >>> >> >>>>>>>>>>> >> > allocated
> > >>> >> >>>>>>>>>>> >> > ==20833==
> > >>> >> >>>>>>>>>>> >> > ==20833== LEAK SUMMARY:
> > >>> >> >>>>>>>>>>> >> > ==20833== definitely lost: 516 bytes in 12
> > blocks
> > >>> >> >>>>>>>>>>> >> > ==20833== indirectly lost: 0 bytes in 0 blocks
> > >>> >> >>>>>>>>>>> >> > ==20833== possibly lost: 33,559,553 bytes in
> > 17
> > >>> >> blocks
> > >>> >> >>>>>>>>>>> >> > ==20833== still reachable: 11,045,257 bytes in
> > 4,990
> > >>> >> blocks
> > >>> >> >>>>>>>>>>> >> > ==20833== suppressed: 0 bytes in 0 blocks
> > >>> >> >>>>>>>>>>> >> > ==20833== Rerun with --leak-check=full to see
> > details
> > >>> of
> > >>> >> leaked memory
> > >>> >> >>>>>>>>>>> >> > ==20833==
> > >>> >> >>>>>>>>>>> >> > ==20833== For counts of detected and suppressed
> > errors,
> > >>> >> rerun with: -v
> > >>> >> >>>>>>>>>>> >> > ==20833== Use --track-origins=yes to see where
> > >>> >> uninitialised values
> > >>> >> >>>>>>>>>>> come
> > >>> >> >>>>>>>>>>> >> > from
> > >>> >> >>>>>>>>>>> >> > ==20833== ERROR SUMMARY: 11 errors from 3
> contexts
> > >>> >> (suppressed: 100
> > >>> >> >>>>>>>>>>> from
> > >>> >> >>>>>>>>>>> >> 9)
> > >>> >> >>>>>>>>>>> >> > 已杀死
> > >>> >> >>>>>>>>>>> >> >
> > >>> >> >>>>>>>>>>> >> > 2015-03-20 23:29 GMT+08:00 singh.janmejay <
> > >>> >> singh.janme...@gmail.com>:
> > >>> >> >>>>>>>>>>> >> >
> > >>> >> >>>>>>>>>>> >> > > Can you please build with debug symbols and
> > repeat
> > >>> the
> > >>> >> valgrind run?
> > >>> >> >>>>>>>>>>> >> > >
> > >>> >> >>>>>>>>>>> >> > > --
> > >>> >> >>>>>>>>>>> >> > > Regards,
> > >>> >> >>>>>>>>>>> >> > > Janmejay
> > >>> >> >>>>>>>>>>> >> > >
> > >>> >> >>>>>>>>>>> >> > > PS: Please blame the typos in this mail on my
> > phone's
> > >>> >> uncivilized
> > >>> >> >>>>>>>>>>> soft
> > >>> >> >>>>>>>>>>> >> > > keyboard sporting it's not-so-smart-assist
> > >>> technology.
> > >>> >> >>>>>>>>>>> >> > >
> > >>> >> >>>>>>>>>>> >> > > On Mar 20, 2015 6:03 PM, "chenlin rao" <
> > >>> >> rao.chen...@gmail.com>
> > >>> >> >>>>>>>>>>> wrote:
> > >>> >> >>>>>>>>>>> >> > >
> > >>> >> >>>>>>>>>>> >> > > > btw: if I change omelasticsearch/omfwd to
> > omfile,
> > >>> >> rsyslogd would
> > >>> >> >>>>>>>>>>> be
> > >>> >> >>>>>>>>>>> >> > > fine...
> > >>> >> >>>>>>>>>>> >> > > >
> > >>> >> >>>>>>>>>>> >> > > > 2015-03-20 20:13 GMT+08:00 chenlin rao <
> > >>> >> rao.chen...@gmail.com>:
> > >>> >> >>>>>>>>>>> >> > > >
> > >>> >> >>>>>>>>>>> >> > > > > 3498.767218405:main Q[DA]:Reg/w0:
> > rainerscript:
> > >>> var
> > >>> >> 200:!msg:
> > >>> >> >>>>>>>>>>> '[ {
> > >>> >> >>>>>>>>>>> >> > > "uid":
> > >>> >> >>>>>>>>>>> >> > > > > "1941604034", "request_header":
> > >>> >> >>>>>>>>>>> >> > > "{\"Accept-Encoding\":\"gzip,deflate\"}",
> > >>> >> >>>>>>>>>>> >> > > > > "network_type": "wifi", "end_time":
> > >>> >> "1426836307406", "dns_ip":
> > >>> >> >>>>>>>>>>> >> > > > > "218.15.203.34,192.168.1.1",
> "response_code":
> > >>> "200",
> > >>> >> >>>>>>>>>>> >> "response_data":
> > >>> >> >>>>>>>>>>> >> > > > "{}",
> > >>> >> >>>>>>>>>>> >> > > > > "start_time": "1426836307328", "act":
> > >>> >> "net_fatal_error", "type":
> > >>> >> >>>>>>>>>>> >> > > > > "net_fatal_error", "request_url":
> "https:\/\/
> > >>> >> api.weibo.cn
> > >>> >> >>>>>>>>>>> >> > > >
> > >>> >> >>>>>>>>>>> >> > >
> > >>> >> >>>>>>>>>>> >> >
> > >>> >> >>>>>>>>>>> >>
> > >>> >> >>>>>>>>>>>
> > >>> >>
> > >>>
> >
> \/2\/client\/url_list?accuracy_level=0&c=android&i=9eef7ba&s=0088f6a4&ua=Xiaomi-MI%202S__weibo__5.0.0__android__android4.4.4&wm=5311_4002&v_f=2&from=1050095010&gsid=4uzH00573m6YZU5oVhYra896c0y&lang=zh_CN&skin=default&oldwm=5311_4002",
> > >>> >> >>>>>>>>>>> >> > > > > "response_status_line": "HTTP\/1.1 200 OK"
> > }, {
> > >>> >> "uid":
> > >>> >> >>>>>>>>>>> >> "1941604034",
> > >>> >> >>>>>>>>>>> >> > > > > "request_header":
> > >>> >> "{\"Accept-Encoding\":\"gzip,deflate\"}",
> > >>> >> >>>>>>>>>>> >> > > > "network_type":
> > >>> >> >>>>>>>>>>> >> > > > > "wifi", "end_time": "1426836320563",
> > "dns_ip":
> > >>> >> >>>>>>>>>>> >> > > > "218.15.203.34,192.168.1.1",
> > >>> >> >>>>>>>>>>> >> > > > > "response_code": "200", "response_data":
> > "{}",
> > >>> >> "start_time":
> > >>> >> >>>>>>>>>>> >> > > > > "1426836320505", "act": "net_fatal_error",
> > >>> "type":
> > >>> >> >>>>>>>>>>> >> "net_fatal_error",
> > >>> >> >>>>>>>>>>> >> > > > > "request_url": "https:\/\/api.weibo.cn
> > >>> >> >>>>>>>>>>> >> > > >
> > >>> >> >>>>>>>>>>> >> > >
> > >>> >> >>>>>>>>>>> >> >
> > >>> >> >>>>>>>>>>> >>
> > >>> >> >>>>>>>>>>>
> > >>> >>
> > >>>
> >
> \/2\/client\/url_list?accuracy_level=0&c=android&i=9eef7ba&s=0088f6a4&ua=Xiaomi-MI%202S__weibo__5.0.0__android__android4.4.4&wm=5311_4002&v_f=2&from=1050095010&gsid=4uzH00573m6YZU5oVhYra896c0y&lang=zh_CN&skin=default&oldwm=5311_4002",
> > >>> >> >>>>>>>>>>> >> > > > > "response_status_line": "HTTP\/1.1 200 OK"
> }
> > ]'
> > >>> >> >>>>>>>>>>> >> > > > > 3498.767338611:main Q[DA]:Reg/w0: eval expr
> > >>> >> 0x62e63a0, return
> > >>> >> >>>>>>>>>>> >> > datatype
> > >>> >> >>>>>>>>>>> >> > > > 'J'
> > >>> >> >>>>>>>>>>> >> > > > > 3498.770732236:main Q[DA]:Reg/w0:
> ACTION
> > 17
> > >>> >> >>>>>>>>>>> >> > > > > [builtin:omfwd:action(type="builtin:omfwd"
> > ...)]
> > >>> >> >>>>>>>>>>> >> > > > > 3498.770963455:main Q[DA]:Reg/w0: executing
> > >>> action
> > >>> >> 17
> > >>> >> >>>>>>>>>>> >> > > > > 3498.771052511:main Q[DA]:Reg/w0: Called
> > action,
> > >>> >> logging to
> > >>> >> >>>>>>>>>>> >> > > builtin:omfwd
> > >>> >> >>>>>>>>>>> >> > > > > ==30138== Thread 3:
> > >>> >> >>>>>>>>>>> >> > > > > ==30138== Invalid read of size 4
> > >>> >> >>>>>>>>>>> >> > > > > ==30138== at 0x4E3E3A0:
> > pthread_mutex_lock (in
> > >>> >> /lib64/
> > >>> >> >>>>>>>>>>> >> > > > > libpthread-2.12.so)
> > >>> >> >>>>>>>>>>> >> > > > > ==30138== by 0x14228D: qqueueEnqMsg (in
> > >>> >> /sbin/rsyslogd)
> > >>> >> >>>>>>>>>>> >> > > > > ==30138== by 0x14BB5B: ??? (in
> > /sbin/rsyslogd)
> > >>> >> >>>>>>>>>>> >> > > > > ==30138== by 0x143C4C: ??? (in
> > /sbin/rsyslogd)
> > >>> >> >>>>>>>>>>> >> > > > > ==30138== by 0x143DAC: ??? (in
> > /sbin/rsyslogd)
> > >>> >> >>>>>>>>>>> >> > > > > ==30138== by 0x143BB9: ??? (in
> > /sbin/rsyslogd)
> > >>> >> >>>>>>>>>>> >> > > > > ==30138== by 0x1442CB: ??? (in
> > /sbin/rsyslogd)
> > >>> >> >>>>>>>>>>> >> > > > > ==30138== by 0x15336B: ??? (in
> > /sbin/rsyslogd)
> > >>> >> >>>>>>>>>>> >> > > > > ==30138== by 0x143112: ??? (in
> > /sbin/rsyslogd)
> > >>> >> >>>>>>>>>>> >> > > > > ==30138== by 0x13D671: wtiWorker (in
> > >>> >> /sbin/rsyslogd)
> > >>> >> >>>>>>>>>>> >> > > > > ==30138== by 0x13D1C1: ??? (in
> > /sbin/rsyslogd)
> > >>> >> >>>>>>>>>>> >> > > > > ==30138== by 0x4E3C9D0: start_thread (in
> > >>> /lib64/
> > >>> >> >>>>>>>>>>> >> > libpthread-2.12.so)
> > >>> >> >>>>>>>>>>> >> > > > > ==30138== Address 0x10 is not stack'd,
> > malloc'd
> > >>> or
> > >>> >> (recently)
> > >>> >> >>>>>>>>>>> >> free'd
> > >>> >> >>>>>>>>>>> >> > > > > ==30138==
> > >>> >> >>>>>>>>>>> >> > > > > ==30138==
> > >>> >> >>>>>>>>>>> >> > > > > ==30138== Process terminating with default
> > action
> > >>> >> of signal 11
> > >>> >> >>>>>>>>>>> >> > > (SIGSEGV)
> > >>> >> >>>>>>>>>>> >> > > > > ==30138== Access not within mapped region
> at
> > >>> >> address 0x10
> > >>> >> >>>>>>>>>>> >> > > > > ==30138== at 0x4E3E3A0:
> > pthread_mutex_lock (in
> > >>> >> /lib64/
> > >>> >> >>>>>>>>>>> >> > > > > libpthread-2.12.so)
> > >>> >> >>>>>>>>>>> >> > > > > ==30138== by 0x14228D: qqueueEnqMsg (in
> > >>> >> /sbin/rsyslogd)
> > >>> >> >>>>>>>>>>> >> > > > > ==30138== by 0x14BB5B: ??? (in
> > /sbin/rsyslogd)
> > >>> >> >>>>>>>>>>> >> > > > > ==30138== by 0x143C4C: ??? (in
> > /sbin/rsyslogd)
> > >>> >> >>>>>>>>>>> >> > > > > ==30138== by 0x143DAC: ??? (in
> > /sbin/rsyslogd)
> > >>> >> >>>>>>>>>>> >> > > > > ==30138== by 0x143BB9: ??? (in
> > /sbin/rsyslogd)
> > >>> >> >>>>>>>>>>> >> > > > > ==30138== by 0x1442CB: ??? (in
> > /sbin/rsyslogd)
> > >>> >> >>>>>>>>>>> >> > > > > ==30138== by 0x15336B: ??? (in
> > /sbin/rsyslogd)
> > >>> >> >>>>>>>>>>> >> > > > > ==30138== by 0x143112: ??? (in
> > /sbin/rsyslogd)
> > >>> >> >>>>>>>>>>> >> > > > > ==30138== by 0x13D671: wtiWorker (in
> > >>> >> /sbin/rsyslogd)
> > >>> >> >>>>>>>>>>> >> > > > > ==30138== by 0x13D1C1: ??? (in
> > /sbin/rsyslogd)
> > >>> >> >>>>>>>>>>> >> > > > > ==30138== by 0x4E3C9D0: start_thread (in
> > >>> /lib64/
> > >>> >> >>>>>>>>>>> >> > libpthread-2.12.so)
> > >>> >> >>>>>>>>>>> >> > > > > ==30138== If you believe this happened as
> a
> > >>> result
> > >>> >> of a stack
> > >>> >> >>>>>>>>>>> >> > > > > ==30138== overflow in your program's main
> > thread
> > >>> >> (unlikely but
> > >>> >> >>>>>>>>>>> >> > > > > ==30138== possible), you can try to
> > increase the
> > >>> >> size of the
> > >>> >> >>>>>>>>>>> >> > > > > ==30138== main thread stack using the
> > >>> >> --main-stacksize= flag.
> > >>> >> >>>>>>>>>>> >> > > > > ==30138== The main thread stack size used
> in
> > >>> this
> > >>> >> run was
> > >>> >> >>>>>>>>>>> >> 10485760.
> > >>> >> >>>>>>>>>>> >> > > > > ==30138==
> > >>> >> >>>>>>>>>>> >> > > > > ==30138== HEAP SUMMARY:
> > >>> >> >>>>>>>>>>> >> > > > > ==30138== in use at exit: 1,078,127,378
> > bytes
> > >>> >> in 291,997
> > >>> >> >>>>>>>>>>> blocks
> > >>> >> >>>>>>>>>>> >> > > > > ==30138== total heap usage: 614,872
> allocs,
> > >>> >> 322,875 frees,
> > >>> >> >>>>>>>>>>> >> > > > 1,586,744,205
> > >>> >> >>>>>>>>>>> >> > > > > bytes allocated
> > >>> >> >>>>>>>>>>> >> > > > > ==30138==
> > >>> >> >>>>>>>>>>> >> > > > > ==30138== LEAK SUMMARY:
> > >>> >> >>>>>>>>>>> >> > > > > ==30138== definitely lost: 33,555,027
> > bytes in
> > >>> >> 13 blocks
> > >>> >> >>>>>>>>>>> >> > > > > ==30138== indirectly lost: 0 bytes in 0
> > blocks
> > >>> >> >>>>>>>>>>> >> > > > > ==30138== possibly lost: 5,760 bytes
> in
> > 18
> > >>> >> blocks
> > >>> >> >>>>>>>>>>> >> > > > > ==30138== still reachable: 1,044,566,591
> > bytes
> > >>> >> in 291,966
> > >>> >> >>>>>>>>>>> blocks
> > >>> >> >>>>>>>>>>> >> > > > > ==30138== suppressed: 0 bytes in 0
> > blocks
> > >>> >> >>>>>>>>>>> >> > > > > ==30138== Rerun with --leak-check=full to
> see
> > >>> >> details of leaked
> > >>> >> >>>>>>>>>>> >> > memory
> > >>> >> >>>>>>>>>>> >> > > > > ==30138==
> > >>> >> >>>>>>>>>>> >> > > > > ==30138== For counts of detected and
> > suppressed
> > >>> >> errors, rerun
> > >>> >> >>>>>>>>>>> with:
> > >>> >> >>>>>>>>>>> >> > -v
> > >>> >> >>>>>>>>>>> >> > > > > ==30138== Use --track-origins=yes to see
> > where
> > >>> >> uninitialised
> > >>> >> >>>>>>>>>>> values
> > >>> >> >>>>>>>>>>> >> > > come
> > >>> >> >>>>>>>>>>> >> > > > > from
> > >>> >> >>>>>>>>>>> >> > > > > ==30138== ERROR SUMMARY: 1809 errors from 5
> > >>> contexts
> > >>> >> >>>>>>>>>>> (suppressed:
> > >>> >> >>>>>>>>>>> >> 131
> > >>> >> >>>>>>>>>>> >> > > > from
> > >>> >> >>>>>>>>>>> >> > > > > 9)
> > >>> >> >>>>>>>>>>> >> > > > > Killed
> > >>> >> >>>>>>>>>>> >> > > > >
> > >>> >> >>>>>>>>>>> >> > > > >
> > >>> >> >>>>>>>>>>> >> > > > > 2015-03-20 8:50 GMT+08:00 singh.janmejay <
> > >>> >> >>>>>>>>>>> singh.janme...@gmail.com
> > >>> >> >>>>>>>>>>> >> >:
> > >>> >> >>>>>>>>>>> >> > > > >
> > >>> >> >>>>>>>>>>> >> > > > >> Can you please run this with valgrind and
> > share
> > >>> >> its output on
> > >>> >> >>>>>>>>>>> >> crash?
> > >>> >> >>>>>>>>>>> >> > > > >>
> > >>> >> >>>>>>>>>>> >> > > > >> --
> > >>> >> >>>>>>>>>>> >> > > > >> Regards,
> > >>> >> >>>>>>>>>>> >> > > > >> Janmejay
> > >>> >> >>>>>>>>>>> >> > > > >>
> > >>> >> >>>>>>>>>>> >> > > > >> PS: Please blame the typos in this mail on
> > my
> > >>> >> phone's
> > >>> >> >>>>>>>>>>> uncivilized
> > >>> >> >>>>>>>>>>> >> > soft
> > >>> >> >>>>>>>>>>> >> > > > >> keyboard sporting it's not-so-smart-assist
> > >>> >> technology.
> > >>> >> >>>>>>>>>>> >> > > > >>
> > >>> >> >>>>>>>>>>> >> > > > >> On Mar 19, 2015 11:10 PM, "chenlin rao" <
> > >>> >> rao.chen...@gmail.com
> > >>> >> >>>>>>>>>>> >
> > >>> >> >>>>>>>>>>> >> > > wrote:
> > >>> >> >>>>>>>>>>> >> > > > >>
> > >>> >> >>>>>>>>>>> >> > > > >> > Hello everyone.
> > >>> >> >>>>>>>>>>> >> > > > >> >
> > >>> >> >>>>>>>>>>> >> > > > >> > I just learnt a foreach syntax from
> > >>> >> >>>>>>>>>>> >> > > `src/tests/json_array_looping.sh`,
> > >>> >> >>>>>>>>>>> >> > > > >> so I
> > >>> >> >>>>>>>>>>> >> > > > >> > try to parse my logdata(yes, long json
> > array
> > >>> in
> > >>> >> msg) as
> > >>> >> >>>>>>>>>>> follow:
> > >>> >> >>>>>>>>>>> >> > > > >> >
> > >>> >> >>>>>>>>>>> >> > > > >> > ```
> > >>> >> >>>>>>>>>>> >> > > > >> >
> > >>> >> >>>>>>>>>>> >> > > > >> > $MaxMessageSize 256k
> > >>> >> >>>>>>>>>>> >> > > > >> >
> > >>> >> >>>>>>>>>>> >> > > > >> > template( name="local6JsonArray"
> > type="string"
> > >>> >> >>>>>>>>>>> >> > string="%$.line%\n" )
> > >>> >> >>>>>>>>>>> >> > > > >> >
> > >>> >> >>>>>>>>>>> >> > > > >> > Ruleset( name="forwardRuleSetJsonArray"
> )
> > >>> >> >>>>>>>>>>> >> > > > >> >
> > >>> >> >>>>>>>>>>> >> > > > >> > {
> > >>> >> >>>>>>>>>>> >> > > > >> >
> > >>> >> >>>>>>>>>>> >> > > > >> > action( type="mmjsonparse" )
> > >>> >> >>>>>>>>>>> >> > > > >> >
> > >>> >> >>>>>>>>>>> >> > > > >> > foreach ($.line in $!msg) do {
> > >>> >> >>>>>>>>>>> >> > > > >> >
> > >>> >> >>>>>>>>>>> >> > > > >> > action ( type="omfile"
> > >>> >> file="/data0/logfile"
> > >>> >> >>>>>>>>>>> >> > > > >> > template="local6JsonArray")
> > >>> >> >>>>>>>>>>> >> > > > >> > }
> > >>> >> >>>>>>>>>>> >> > > > >> > }
> > >>> >> >>>>>>>>>>> >> > > > >> > ```
> > >>> >> >>>>>>>>>>> >> > > > >> >
> > >>> >> >>>>>>>>>>> >> > > > >> > But I always got segment fault after
> > restarted
> > >>> >> few minutes.
> > >>> >> >>>>>>>>>>> >> > > > >> >
> > >>> >> >>>>>>>>>>> >> > > > >> > Before the rsyslogd died , I can watch
> > some
> > >>> >> correct lines in
> > >>> >> >>>>>>>>>>> the
> > >>> >> >>>>>>>>>>> >> > > > >> > "/data0/logfile".
> > >>> >> >>>>>>>>>>> >> > > > >> >
> > >>> >> >>>>>>>>>>> >> > > > >> > While running `rsyslogd -dn`, I watch
> the
> > last
> > >>> >> five lines
> > >>> >> >>>>>>>>>>> were:
> > >>> >> >>>>>>>>>>> >> > > > >> >
> > >>> >> >>>>>>>>>>> >> > > > >> > ```
> > >>> >> >>>>>>>>>>> >> > > > >> >
> > >>> >> >>>>>>>>>>> >> > > > >> > 3295.820218609:main Q:Reg/w0 :
> > FOREACH
> > >>> >> .line IN
> > >>> >> >>>>>>>>>>> >> > > > >> >
> > >>> >> >>>>>>>>>>> >> > > > >> > 3295.820227632:main Q:Reg/w0 :
> var
> > >>> '!msg'
> > >>> >> >>>>>>>>>>> >> > > > >> >
> > >>> >> >>>>>>>>>>> >> > > > >> > 3295.820240109:main Q:Reg/w0 : eval
> expr
> > >>> >> 0x7fe2044fefa0,
> > >>> >> >>>>>>>>>>> type
> > >>> >> >>>>>>>>>>> >> > > 'V[86]'
> > >>> >> >>>>>>>>>>> >> > > > >> >
> > >>> >> >>>>>>>>>>> >> > > > >> > 3295.821747011:main Q:Reg/w0 :
> > rainerscript:
> > >>> >> var 200:!msg:
> > >>> >> >>>>>>>>>>> >> > > '{"msg":[{
> > >>> >> >>>>>>>>>>> >> > > > >> >
> > "end_time":"142...longmsghere....\"verieval
> > >>> expr
> > >>> >> >>>>>>>>>>> 0x7fe2044fefa0,
> > >>> >> >>>>>>>>>>> >> > > > return
> > >>> >> >>>>>>>>>>> >> > > > >> > datatype 'J'
> > >>> >> >>>>>>>>>>> >> > > > >> > ```
> > >>> >> >>>>>>>>>>> >> > > > >> >
> > >>> >> >>>>>>>>>>> >> > > > >> > The length of such line is about
> 20k~30k,
> > far
> > >>> >> away below my
> > >>> >> >>>>>>>>>>> >> > > > >> > $MaxMessageSize.
> > >>> >> >>>>>>>>>>> >> > > > >> >
> > >>> >> >>>>>>>>>>> >> > > > >> > So, why rsyslogd segment fault?
> > >>> >> >>>>>>>>>>> >> > > > >> >
> > >>> _______________________________________________
> > >>> >> >>>>>>>>>>> >> > > > >> > rsyslog mailing list
> > >>> >> >>>>>>>>>>> >> > > > >> >
> > >>> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> > >>> >> >>>>>>>>>>> >> > > > >> >
> > http://www.rsyslog.com/professional-services/
> > >>> >> >>>>>>>>>>> >> > > > >> > What's up with rsyslog? Follow
> > >>> >> https://twitter.com/rgerhards
> > >>> >> >>>>>>>>>>> >> > > > >> > NOTE WELL: This is a PUBLIC mailing
> list,
> > >>> posts
> > >>> >> are ARCHIVED
> > >>> >> >>>>>>>>>>> by
> > >>> >> >>>>>>>>>>> >> a
> > >>> >> >>>>>>>>>>> >> > > > myriad
> > >>> >> >>>>>>>>>>> >> > > > >> > of sites beyond our control. PLEASE
> > >>> UNSUBSCRIBE
> > >>> >> and DO NOT
> > >>> >> >>>>>>>>>>> POST
> > >>> >> >>>>>>>>>>> >> if
> > >>> >> >>>>>>>>>>> >> > > you
> > >>> >> >>>>>>>>>>> >> > > > >> > DON'T LIKE THAT.
> > >>> >> >>>>>>>>>>> >> > > > >> >
> > >>> >> >>>>>>>>>>> >> > > > >>
> > _______________________________________________
> > >>> >> >>>>>>>>>>> >> > > > >> rsyslog mailing list
> > >>> >> >>>>>>>>>>> >> > > > >>
> > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> > >>> >> >>>>>>>>>>> >> > > > >>
> > http://www.rsyslog.com/professional-services/
> > >>> >> >>>>>>>>>>> >> > > > >> What's up with rsyslog? Follow
> > >>> >> https://twitter.com/rgerhards
> > >>> >> >>>>>>>>>>> >> > > > >> NOTE WELL: This is a PUBLIC mailing list,
> > posts
> > >>> >> are ARCHIVED
> > >>> >> >>>>>>>>>>> by a
> > >>> >> >>>>>>>>>>> >> > > myriad
> > >>> >> >>>>>>>>>>> >> > > > >> of sites beyond our control. PLEASE
> > UNSUBSCRIBE
> > >>> >> and DO NOT
> > >>> >> >>>>>>>>>>> POST if
> > >>> >> >>>>>>>>>>> >> > you
> > >>> >> >>>>>>>>>>> >> > > > >> DON'T LIKE THAT.
> > >>> >> >>>>>>>>>>> >> > > > >>
> > >>> >> >>>>>>>>>>> >> > > > >
> > >>> >> >>>>>>>>>>> >> > > > >
> > >>> >> >>>>>>>>>>> >> > > >
> _______________________________________________
> > >>> >> >>>>>>>>>>> >> > > > rsyslog mailing list
> > >>> >> >>>>>>>>>>> >> > > >
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > >>> >> >>>>>>>>>>> >> > > >
> http://www.rsyslog.com/professional-services/
> > >>> >> >>>>>>>>>>> >> > > > What's up with rsyslog? Follow
> > >>> >> https://twitter.com/rgerhards
> > >>> >> >>>>>>>>>>> >> > > > NOTE WELL: This is a PUBLIC mailing list,
> > posts are
> > >>> >> ARCHIVED by a
> > >>> >> >>>>>>>>>>> >> > myriad
> > >>> >> >>>>>>>>>>> >> > > > of sites beyond our control. PLEASE
> > UNSUBSCRIBE and
> > >>> >> DO NOT POST if
> > >>> >> >>>>>>>>>>> >> you
> > >>> >> >>>>>>>>>>> >> > > > DON'T LIKE THAT.
> > >>> >> >>>>>>>>>>> >> > > >
> > >>> >> >>>>>>>>>>> >> > > _______________________________________________
> > >>> >> >>>>>>>>>>> >> > > rsyslog mailing list
> > >>> >> >>>>>>>>>>> >> > >
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > >>> >> >>>>>>>>>>> >> > > http://www.rsyslog.com/professional-services/
> > >>> >> >>>>>>>>>>> >> > > What's up with rsyslog? Follow
> > >>> >> https://twitter.com/rgerhards
> > >>> >> >>>>>>>>>>> >> > > NOTE WELL: This is a PUBLIC mailing list, posts
> > are
> > >>> >> ARCHIVED by a
> > >>> >> >>>>>>>>>>> >> myriad
> > >>> >> >>>>>>>>>>> >> > > of sites beyond our control. PLEASE UNSUBSCRIBE
> > and
> > >>> DO
> > >>> >> NOT POST if
> > >>> >> >>>>>>>>>>> you
> > >>> >> >>>>>>>>>>> >> > > DON'T LIKE THAT.
> > >>> >> >>>>>>>>>>> >> > >
> > >>> >> >>>>>>>>>>> >> > _______________________________________________
> > >>> >> >>>>>>>>>>> >> > rsyslog mailing list
> > >>> >> >>>>>>>>>>> >> >
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> > >>> >> >>>>>>>>>>> >> > http://www.rsyslog.com/professional-services/
> > >>> >> >>>>>>>>>>> >> > What's up with rsyslog? Follow
> > >>> >> https://twitter.com/rgerhards
> > >>> >> >>>>>>>>>>> >> > NOTE WELL: This is a PUBLIC mailing list, posts
> are
> > >>> >> ARCHIVED by a
> > >>> >> >>>>>>>>>>> myriad
> > >>> >> >>>>>>>>>>> >> > of sites beyond our control. PLEASE UNSUBSCRIBE
> > and DO
> > >>> >> NOT POST if you
> > >>> >> >>>>>>>>>>> >> > DON'T LIKE THAT.
> > >>> >> >>>>>>>>>>> >> _______________________________________________
> > >>> >> >>>>>>>>>>> >> rsyslog mailing list
> > >>> >> >>>>>>>>>>> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> > >>> >> >>>>>>>>>>> >> http://www.rsyslog.com/professional-services/
> > >>> >> >>>>>>>>>>> >> What's up with rsyslog? Follow
> > >>> >> https://twitter.com/rgerhards
> > >>> >> >>>>>>>>>>> >> NOTE WELL: This is a PUBLIC mailing list, posts are
> > >>> >> ARCHIVED by a myriad
> > >>> >> >>>>>>>>>>> >> of sites beyond our control. PLEASE UNSUBSCRIBE and
> > DO
> > >>> NOT
> > >>> >> POST if you
> > >>> >> >>>>>>>>>>> >> DON'T LIKE THAT.
> > >>> >> >>>>>>>>>>> >>
> > >>> >> >>>>>>>>>>> > _______________________________________________
> > >>> >> >>>>>>>>>>> > rsyslog mailing list
> > >>> >> >>>>>>>>>>> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > >>> >> >>>>>>>>>>> > http://www.rsyslog.com/professional-services/
> > >>> >> >>>>>>>>>>> > What's up with rsyslog? Follow
> > >>> https://twitter.com/rgerhards
> > >>> >> >>>>>>>>>>> > NOTE WELL: This is a PUBLIC mailing list, posts are
> > >>> ARCHIVED
> > >>> >> by a myriad
> > >>> >> >>>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO
> > NOT
> > >>> >> POST if you
> > >>> >> >>>>>>>>>>> DON'T LIKE THAT.
> > >>> >> >>>>>>>>>>> _______________________________________________
> > >>> >> >>>>>>>>>>> rsyslog mailing list
> > >>> >> >>>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> > >>> >> >>>>>>>>>>> http://www.rsyslog.com/professional-services/
> > >>> >> >>>>>>>>>>> What's up with rsyslog? Follow
> > >>> https://twitter.com/rgerhards
> > >>> >> >>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are
> > ARCHIVED
> > >>> >> by a myriad
> > >>> >> >>>>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO
> > NOT
> > >>> >> POST if you
> > >>> >> >>>>>>>>>>> DON'T LIKE THAT.
> > >>> >> >>>>>>>>>>>
> > >>> >> >>>>>>>>>> _______________________________________________
> > >>> >> >>>>>>>>>> rsyslog mailing list
> > >>> >> >>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> > >>> >> >>>>>>>>>> http://www.rsyslog.com/professional-services/
> > >>> >> >>>>>>>>>> What's up with rsyslog? Follow
> > https://twitter.com/rgerhards
> > >>> >> >>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are
> > ARCHIVED
> > >>> by
> > >>> >> a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO
> NOT
> > >>> POST if
> > >>> >> you DON'T LIKE THAT.
> > >>> >> >>>>>>>>>
> > >>> >> >>>>>>>>> _______________________________________________
> > >>> >> >>>>>>>>> rsyslog mailing list
> > >>> >> >>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> > >>> >> >>>>>>>>> http://www.rsyslog.com/professional-services/
> > >>> >> >>>>>>>>> What's up with rsyslog? Follow
> > https://twitter.com/rgerhards
> > >>> >> >>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are
> > ARCHIVED
> > >>> by
> > >>> >> a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO
> NOT
> > >>> POST if
> > >>> >> you DON'T LIKE THAT.
> > >>> >> >>>>>>>>
> > >>> >> >>>>>>>>
> > >>> >> >>>>>>>>
> > >>> >> >>>>>>>> --
> > >>> >> >>>>>>>> Regards,
> > >>> >> >>>>>>>> Janmejay
> > >>> >> >>>>>>>> http://codehunk.wordpress.com
> > >>> >> >>>>>>>> _______________________________________________
> > >>> >> >>>>>>>> rsyslog mailing list
> > >>> >> >>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> > >>> >> >>>>>>>> http://www.rsyslog.com/professional-services/
> > >>> >> >>>>>>>> What's up with rsyslog? Follow
> > https://twitter.com/rgerhards
> > >>> >> >>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are
> > ARCHIVED
> > >>> by a
> > >>> >> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
> > POST
> > >>> if
> > >>> >> you DON'T LIKE THAT.
> > >>> >> >>>>>>> _______________________________________________
> > >>> >> >>>>>>> rsyslog mailing list
> > >>> >> >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> > >>> >> >>>>>>> http://www.rsyslog.com/professional-services/
> > >>> >> >>>>>>> What's up with rsyslog? Follow
> > https://twitter.com/rgerhards
> > >>> >> >>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are
> > ARCHIVED by
> > >>> a
> > >>> >> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
> > POST
> > >>> if
> > >>> >> you DON'T LIKE THAT.
> > >>> >> >>>>>>
> > >>> >> >>>>>>
> > >>> >> >>>>>>
> > >>> >> >>>>>> --
> > >>> >> >>>>>> Regards,
> > >>> >> >>>>>> Janmejay
> > >>> >> >>>>>> http://codehunk.wordpress.com
> > >>> >> >>>>>
> > >>> >> >>>>>
> > >>> >> >>>>>
> > >>> >> >>>>> --
> > >>> >> >>>>> Regards,
> > >>> >> >>>>> Janmejay
> > >>> >> >>>>> http://codehunk.wordpress.com
> > >>> >> >>>>> _______________________________________________
> > >>> >> >>>>> rsyslog mailing list
> > >>> >> >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> > >>> >> >>>>> http://www.rsyslog.com/professional-services/
> > >>> >> >>>>> What's up with rsyslog? Follow
> https://twitter.com/rgerhards
> > >>> >> >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
> > by a
> > >>> >> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
> > POST
> > >>> if
> > >>> >> you DON'T LIKE THAT.
> > >>> >> >>>> _______________________________________________
> > >>> >> >>>> rsyslog mailing list
> > >>> >> >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> > >>> >> >>>> http://www.rsyslog.com/professional-services/
> > >>> >> >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
> > >>> >> >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
> > by a
> > >>> >> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
> > POST
> > >>> if
> > >>> >> you DON'T LIKE THAT.
> > >>> >> >>>
> > >>> >> >>>
> > >>> >> >>>
> > >>> >> >>> --
> > >>> >> >>> Regards,
> > >>> >> >>> Janmejay
> > >>> >> >>> http://codehunk.wordpress.com
> > >>> >> >>
> > >>> >> >>
> > >>> >> >>
> > >>> >> >> --
> > >>> >> >> Regards,
> > >>> >> >> Janmejay
> > >>> >> >> http://codehunk.wordpress.com
> > >>> >> >
> > >>> >> >
> > >>> >> >
> > >>> >> > --
> > >>> >> > Regards,
> > >>> >> > Janmejay
> > >>> >> > http://codehunk.wordpress.com
> > >>> >>
> > >>> >>
> > >>> >>
> > >>> >> --
> > >>> >> Regards,
> > >>> >> Janmejay
> > >>> >> http://codehunk.wordpress.com
> > >>> >> _______________________________________________
> > >>> >> rsyslog mailing list
> > >>> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> > >>> >> http://www.rsyslog.com/professional-services/
> > >>> >> What's up with rsyslog? Follow https://twitter.com/rgerhards
> > >>> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> > myriad
> > >>> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
> > you
> > >>> >> DON'T LIKE THAT.
> > >>> >>
> > >>> > _______________________________________________
> > >>> > rsyslog mailing list
> > >>> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > >>> > http://www.rsyslog.com/professional-services/
> > >>> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > >>> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> > myriad
> > >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
> you
> > >>> DON'T LIKE THAT.
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> Regards,
> > >>> Janmejay
> > >>> http://codehunk.wordpress.com
> > >>> _______________________________________________
> > >>> rsyslog mailing list
> > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> > >>> http://www.rsyslog.com/professional-services/
> > >>> What's up with rsyslog? Follow https://twitter.com/rgerhards
> > >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> > myriad
> > >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
> you
> > >>> DON'T LIKE THAT.
> > >>>
> > >> _______________________________________________
> > >> rsyslog mailing list
> > >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> > >> http://www.rsyslog.com/professional-services/
> > >> What's up with rsyslog? Follow https://twitter.com/rgerhards
> > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
> > you DON'T LIKE THAT.
> > >
> > >
> > >
> > > --
> > > Regards,
> > > Janmejay
> > > http://codehunk.wordpress.com
> >
> >
> >
> > --
> > Regards,
> > Janmejay
> > http://codehunk.wordpress.com
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> > DON'T LIKE THAT.
> >
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
