I check the `}]` end just because there are some too long lines( 20MB+)
might be transcated. Those line would cause mmjsonparse crash. I think this
is also a place we can improve, but it's another question.
2015-04-02 14:48 GMT+08:00 chenlin rao <rao.chen...@gmail.com>:
> @cee:{"msg":[{"content":"Error Domain=WBBussinessErrorDomain Code=20030
> \"the user base info is not exists!\" UserInfo=0x17127cc00
> {NSLocalizedDescription=the user base info is not exists!,
> WBBussinessErrorInfoKey=<CFBasicHash 0x17127cb40 [0x193536c80]>{type =
> immutable dict, count = 4,\nentries =>\n\t0 : <CFString 0x170c38e20
> [0x193536c80]>{contents = \"errno\"} = <CFNumber 0xb00000000004e3e3
> [0x193536c80]>{value = +20030, type = kCFNumberSInt64Type}\n\t4 : <CFString
> 0x170c38e60 [0x193536c80]>{contents = \"errtype\"} = <CFString 0x170c38dc0
> [0x193536c80]>{contents = \"DEFAULT_ERROR\"}\n\t5 : <CFString 0x170c38de0
> [0x193536c80]>{contents = \"errmsg\"} = <CFString 0x17127cb00
> [0x193536c80]>{contents = \"the user base info is not exists!\"}\n\t6 :
> <CFString 0x170c38d60 [0x193536c80]>{contents = \"isblock\"} = <CFBoolean
> 0x193537030 [0x193536c80]>{value =
> false}\n}\n}","__date":1427956340.3598,"dns_ip":"172.27.35.1 202.103.44.150
> ","start_time":"1427956340.237840","response_header":{"Content-Type":"application\/json;charset=UTF-8","SINA-TS":"MzdjNmMzNjggMCAwIDAgNyAxNQo=","X-Powered-By":"PHP\/5.4.11","Content-Encoding":"gzip","Server":"nginx\/1.6.1","X-Error-Code":"20030","Transfer-Encoding":"Identity","SINA-LB":"aGEuMjEwLmcxLnlmLmxiLnNpbmFub2RlLmNvbQ==","Date":"Thu,
> 02 Apr 2015 06:32:17
> GMT","Connection":"keep-alive","X-Log-Uid":"2255484080","Vary":"Accept-Encoding","PROC_NODE":"
> web232.mweibo.yf.sinanode.com
> "},"type":"bussiness_error","request_method":"GET","request_url":"https:\/\/
> api.weibo.cn\/2\/health\/user_show?gsid=4usj7d103XGQMsvwx1cot9sKM1i&wm=3333_2001&i=4739d26&b=1&from=1052093010&c=iphone&v_p=18&skin=default&v_f=1&s=a71b07ec&lang=zh_CN&ua=iPhone7,1__weibo__5.2.0__iphone__os8.1.2&lfid=1005052255484080&luicode=10000011&lcardid=1005052255484080_-_WEIBO_INDEX_PROFILE_SPORT&sourcetype=page&uicode=10000304","network_type":"wifi","response_data":"{\"errmsg\":\"the
> user base info is not
> exists!\",\"errno\":20030,\"errtype\":\"DEFAULT_ERROR\",\"isblock\":false}","end_time":"1427956340.352432","request_header":{"User-Agent":"Weibo\/42150000
> (iPhone; iOS 8.1.2;
> Scale\/3.00)","X-Log-Uid":"2255484080","SNRT":"normal"}}]}
>
>
>
> 2015-04-02 13:35 GMT+08:00 singh.janmejay <singh.janme...@gmail.com>:
>
>> Hadn't noticed the use of msg key early-on, so "]}" shouldn't be a
>> problem, but I can't reproduce it with input i crafted to work with
>> it.
>> Here is the crafted input:
>> @cee:{"msg" : ["G\"foo\":\"10\"}", "G\"bar\":\"20\""]}
>>
>> This is what does fail it, im using a bad type (it doesn't do type-check
>> yet):
>> @cee:{"msg" : "G\"foo\":\"10\"}", "bar" : ["foo"]}
>>
>> Here it ending in value of 'bar' satisfies the ']}' conditional, but
>> msg isn't an array. I'll add array-typecheck, but other than this I
>> have been unable to reproduce it, so this may not fix your problem.
>>
>> I think your input line (or entire input file) will help.
>>
>> On Thu, Apr 2, 2015 at 9:39 AM, singh.janmejay <singh.janme...@gmail.com>
>> wrote:
>> > Hi Chenlin,
>> >
>> > I finally have some time to look at it. (Sorry for the delay.)
>> >
>> > Can you please share a sample log-line? Feel free to anonymize it, but
>> > please keep the structure and types as is.
>> >
>> > I see that you are checking $msg ends with "]}", that means it is an
>> > object, not an array.
>> >
>> > If my interpretation is correct, trying to iterate over object instead
>> > of array may be causing this problem. (I haven't looked at code yet to
>> > verify this manifestation though).
>> >
>> > On Tue, Mar 31, 2015 at 2:32 PM, Rainer Gerhards
>> > <rgerha...@hq.adiscon.com> wrote:
>> >> 2015-03-30 19:55 GMT+02:00 singh.janmejay <singh.janme...@gmail.com>:
>> >>> Sorry, I haven't had a chance to look at it yet.
>> >>
>> >> Take your time, I know how challenging it is at times ;) I just wanted
>> >> to spread the good news that we have a good repro.
>> >>
>> >> Rainer
>> >>>
>> >>> Will get to this asap.
>> >>>
>> >>> On Mon, Mar 30, 2015 at 2:36 PM, Rainer Gerhards
>> >>> <rgerha...@hq.adiscon.com> wrote:
>> >>>> Co-incidentally, an unrelated testbench run also experienced problems
>> >>>> in this area. I have attached the testbench log.
>> >>>>
>> >>>> Here is the valgrind report:
>> >>>>
>> >>>> 5812.499032607:main Q:Reg/w0 : wti 0x614e9e0: we need to create a
>> new
>> >>>> action worker instance for action 2
>> >>>>
>> >>>> 5812.499181273:main Q:Reg/w0 : Action 2 transitioned to state: itx
>> >>>>
>> >>>> 5812.499314576:main Q:Reg/w0 : FOREACH .corge IN
>> >>>>
>> >>>> 5812.499625664:main Q:Reg/w0 : var '.quux!bar'
>> >>>>
>> >>>> 5812.500078770:main Q:Reg/w0 : eval expr 0x60849e0, type 'V[86]'
>> >>>>
>> >>>> ==2284== Thread 6:
>> >>>>
>> >>>> ==2284== Invalid read of size 8
>> >>>>
>> >>>> ==2284== at 0x587B134: lh_table_lookup_entry (in
>> >>>> /usr/lib/x86_64-linux-gnu/libjson.so.0.0.1)
>> >>>>
>> >>>> ==2284== by 0x587B1A8: lh_table_lookup (in
>> >>>> /usr/lib/x86_64-linux-gnu/libjson.so.0.0.1)
>> >>>>
>> >>>> ==2284== by 0x416F98: jsonVarExtract (msg.c:4218)
>> >>>>
>> >>>> ==2284== by 0x41C82F: msgGetJSONPropJSON (msg.c:2848)
>> >>>>
>> >>>> ==2284== by 0x4562D0: cnfexprEval (rainerscript.c:1785)
>> >>>>
>> >>>> ==2284== by 0x45842E: cnfexprEvalCollection (rainerscript.c:2483)
>> >>>>
>> >>>> ==2284== by 0x435CFC: scriptExec (ruleset.c:271)
>> >>>>
>> >>>> ==2284== by 0x435D6B: scriptExec (ruleset.c:284)
>> >>>>
>> >>>> ==2284== by 0x436233: processBatch (ruleset.c:503)
>> >>>>
>> >>>> ==2284== by 0x44464B: msgConsumer (rsyslogd.c:576)
>> >>>>
>> >>>> ==2284== by 0x432AC2: ConsumerReg (queue.c:1897)
>> >>>>
>> >>>> ==2284== by 0x42F55E: wtiWorker (wti.c:334)
>> >>>>
>> >>>> ==2284== Address 0x61645c8 is 8 bytes before a block of size 5
>> alloc'd
>> >>>>
>> >>>> ==2284== at 0x4C2B6CD: malloc (in
>> >>>> /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
>> >>>>
>> >>>> ==2284== by 0x5D0B081: strdup (strdup.c:43)
>> >>>>
>> >>>> ==2284== by 0x5878A5C: json_object_object_add (in
>> >>>> /usr/lib/x86_64-linux-gnu/libjson.so.0.0.1)
>> >>>>
>> >>>> ==2284== by 0x41E419: msgAddJSON (msg.c:4379)
>> >>>>
>> >>>> ==2284== by 0x41F2E0: msgSetJSONFromVar (msg.c:4554)
>> >>>>
>> >>>> ==2284== by 0x435D4E: scriptExec (ruleset.c:283)
>> >>>>
>> >>>> ==2284== by 0x436233: processBatch (ruleset.c:503)
>> >>>>
>> >>>> ==2284== by 0x44464B: msgConsumer (rsyslogd.c:576)
>> >>>>
>> >>>> ==2284== by 0x432AC2: ConsumerReg (queue.c:1897)
>> >>>>
>> >>>> ==2284== by 0x42F55E: wtiWorker (wti.c:334)
>> >>>>
>> >>>> ==2284== by 0x42E6C6: wtpWorker (wtp.c:389)
>> >>>>
>> >>>> ==2284== by 0x5050E99: start_thread (pthread_create.c:308)
>> >>>>
>> >>>> ==2284==
>> >>>>
>> >>>> ==2284== Jump to the invalid address stated on the next line
>> >>>>
>> >>>> ==2284== at 0x0: ???
>> >>>>
>> >>>> ==2284== by 0x587B136: lh_table_lookup_entry (in
>> >>>> /usr/lib/x86_64-linux-gnu/libjson.so.0.0.1)
>> >>>>
>> >>>> ==2284== by 0x587B1A8: lh_table_lookup (in
>> >>>> /usr/lib/x86_64-linux-gnu/libjson.so.0.0.1)
>> >>>>
>> >>>> ==2284== by 0x416F98: jsonVarExtract (msg.c:4218)
>> >>>>
>> >>>> ==2284== by 0x41C82F: msgGetJSONPropJSON (msg.c:2848)
>> >>>>
>> >>>> ==2284== by 0x4562D0: cnfexprEval (rainerscript.c:1785)
>> >>>>
>> >>>> ==2284== by 0x45842E: cnfexprEvalCollection (rainerscript.c:2483)
>> >>>>
>> >>>> ==2284== by 0x435CFC: scriptExec (ruleset.c:271)
>> >>>>
>> >>>> ==2284== by 0x435D6B: scriptExec (ruleset.c:284)
>> >>>>
>> >>>> ==2284== by 0x436233: processBatch (ruleset.c:503)
>> >>>>
>> >>>> ==2284== by 0x44464B: msgConsumer (rsyslogd.c:576)
>> >>>>
>> >>>> ==2284== by 0x432AC2: ConsumerReg (queue.c:1897)
>> >>>>
>> >>>> ==2284== Address 0x0 is not stack'd, malloc'd or (recently) free'd
>> >>>>
>> >>>> ==2284==
>> >>>>
>> >>>> ==2284==
>> >>>>
>> >>>> ==2284== Process terminating with default action of signal 11
>> (SIGSEGV)
>> >>>>
>> >>>> ==2284== Bad permissions for mapped region at address 0x0
>> >>>>
>> >>>> ==2284== at 0x0: ???
>> >>>>
>> >>>> ==2284== by 0x587B136: lh_table_lookup_entry (in
>> >>>> /usr/lib/x86_64-linux-gnu/libjson.so.0.0.1)
>> >>>>
>> >>>> ==2284== by 0x587B1A8: lh_table_lookup (in
>> >>>> /usr/lib/x86_64-linux-gnu/libjson.so.0.0.1)
>> >>>>
>> >>>> ==2284== by 0x416F98: jsonVarExtract (msg.c:4218)
>> >>>>
>> >>>> ==2284== by 0x41C82F: msgGetJSONPropJSON (msg.c:2848)
>> >>>>
>> >>>> ==2284== by 0x4562D0: cnfexprEval (rainerscript.c:1785)
>> >>>>
>> >>>> ==2284== by 0x45842E: cnfexprEvalCollection (rainerscript.c:2483)
>> >>>>
>> >>>> ==2284== by 0x435CFC: scriptExec (ruleset.c:271)
>> >>>>
>> >>>> ==2284== by 0x435D6B: scriptExec (ruleset.c:284)
>> >>>>
>> >>>> ==2284== by 0x436233: processBatch (ruleset.c:503)
>> >>>>
>> >>>> ==2284== by 0x44464B: msgConsumer (rsyslogd.c:576)
>> >>>>
>> >>>> ==2284== by 0x432AC2: ConsumerReg (queue.c:1897)
>> >>>>
>> >>>> ==2284==
>> >>>>
>> >>>> ==2284== HEAP SUMMARY:
>> >>>>
>> >>>> ==2284== in use at exit: 961,978 bytes in 1,632 blocks
>> >>>>
>> >>>> ==2284== total heap usage: 2,448 allocs, 816 frees, 1,023,184
>> bytes allocated
>> >>>>
>> >>>> ==2284==
>> >>>>
>> >>>> ==2284== LEAK SUMMARY:
>> >>>>
>> >>>> ==2284== definitely lost: 6 bytes in 1 blocks
>> >>>>
>> >>>> ==2284== indirectly lost: 0 bytes in 0 blocks
>> >>>>
>> >>>> ==2284== possibly lost: 2,592 bytes in 9 blocks
>> >>>>
>> >>>> ==2284== still reachable: 959,380 bytes in 1,622 blocks
>> >>>>
>> >>>> ==2284== suppressed: 0 bytes in 0 blocks
>> >>>>
>> >>>> ==2284== Rerun with --leak-check=full to see details of leaked memory
>> >>>>
>> >>>> ==2284==
>> >>>>
>> >>>> ==2284== For counts of detected and suppressed errors, rerun with: -v
>> >>>>
>> >>>> ==2284== ERROR SUMMARY: 2 errors from 2 contexts (suppressed: 2 from
>> 2)
>> >>>>
>> >>>> receiving response: Connection reset by peer
>> >>>>
>> >>>> ./diag.sh: line 17: 2284 Killed $valgrind
>> >>>> ../tools/rsyslogd -C -n -irsyslog$3.pid -M../runtime/.libs:../.libs
>> >>>> -f$srcdir/testsuites/$2
>> >>>>
>> >>>> FAIL: json_array_looping.sh
>> >>>>
>> >>>> Rainer
>> >>>>
>> >>>> 2015-03-21 17:50 GMT+01:00 chenlin rao <rao.chen...@gmail.com>:
>> >>>>> thanks for the information. But rsyslogd also fault after I change
>> >>>>> mmjsonparse action config as `action ( type="mmjsonparse"
>> >>>>> name="action_jsonarray-parse"`. output as follow:
>> >>>>>
>> >>>>> 6008.997308131:main Q:Reg/w1 : eval expr 0x52f18f0, return
>> datatype 'J'
>> >>>>>
>> >>>>> 6009.002050327:main Q:Reg/w1 : ACTION 16
>> >>>>> [builtin:omfwd:action(type="builtin:omfwd" ...)]
>> >>>>>
>> >>>>> 6009.002357554:main Q:Reg/w1 : executing action 16
>> >>>>>
>> >>>>> 6009.002454575:main Q:Reg/w1 : Called action, logging to
>> builtin:omfwd
>> >>>>>
>> >>>>> ==9947== Thread 22:
>> >>>>>
>> >>>>> ==9947== Invalid read of size 4
>> >>>>>
>> >>>>> ==9947== at 0x3A044093A0: pthread_mutex_lock (in /lib64/
>> >>>>> libpthread-2.12.so)
>> >>>>>
>> >>>>> ==9947== by 0x438053: qqueueEnqMsg (queue.c:2856)
>> >>>>>
>> >>>>> ==9947== by 0x4421AB: doSubmitToActionQ (action.c:1403)
>> >>>>>
>> >>>>> ==9947== by 0x439D6C: scriptExec (ruleset.c:197)
>> >>>>>
>> >>>>> ==9947== by 0x439ECC: scriptExec (ruleset.c:284)
>> >>>>>
>> >>>>> ==9947== by 0x439CD9: scriptExec (ruleset.c:416)
>> >>>>>
>> >>>>> ==9947== by 0x439CD9: scriptExec (ruleset.c:416)
>> >>>>>
>> >>>>> ==9947== by 0x439CD9: scriptExec (ruleset.c:416)
>> >>>>>
>> >>>>> ==9947== by 0x43A40B: processBatch (ruleset.c:503)
>> >>>>>
>> >>>>> ==9947== by 0x44A403: msgConsumer (rsyslogd.c:575)
>> >>>>>
>> >>>>> ==9947== by 0x438D12: ConsumerReg (queue.c:1897)
>> >>>>>
>> >>>>> ==9947== by 0x43314B: wtiWorker (wti.c:334)
>> >>>>>
>> >>>>> ==9947== Address 0x10 is not stack'd, malloc'd or (recently) free'd
>> >>>>>
>> >>>>> ==9947==
>> >>>>>
>> >>>>> ==9947==
>> >>>>>
>> >>>>> ==9947== Process terminating with default action of signal 11
>> (SIGSEGV)
>> >>>>>
>> >>>>> ==9947== Access not within mapped region at address 0x10
>> >>>>>
>> >>>>> ==9947== at 0x3A044093A0: pthread_mutex_lock (in /lib64/
>> >>>>> libpthread-2.12.so)
>> >>>>>
>> >>>>> ==9947== by 0x438053: qqueueEnqMsg (queue.c:2856)
>> >>>>>
>> >>>>> ==9947== by 0x4421AB: doSubmitToActionQ (action.c:1403)
>> >>>>>
>> >>>>> ==9947== by 0x439D6C: scriptExec (ruleset.c:197)
>> >>>>>
>> >>>>> ==9947== by 0x439ECC: scriptExec (ruleset.c:284)
>> >>>>>
>> >>>>> ==9947== by 0x439CD9: scriptExec (ruleset.c:416)
>> >>>>>
>> >>>>> ==9947== by 0x439CD9: scriptExec (ruleset.c:416)
>> >>>>>
>> >>>>> ==9947== by 0x439CD9: scriptExec (ruleset.c:416)
>> >>>>>
>> >>>>> ==9947== by 0x43A40B: processBatch (ruleset.c:503)
>> >>>>>
>> >>>>> ==9947== by 0x44A403: msgConsumer (rsyslogd.c:575)
>> >>>>>
>> >>>>> ==9947== by 0x438D12: ConsumerReg (queue.c:1897)
>> >>>>>
>> >>>>> ==9947== by 0x43314B: wtiWorker (wti.c:334)
>> >>>>>
>> >>>>> ==9947== If you believe this happened as a result of a stack
>> >>>>>
>> >>>>> ==9947== overflow in your program's main thread (unlikely but
>> >>>>>
>> >>>>> ==9947== possible), you can try to increase the size of the
>> >>>>>
>> >>>>> ==9947== main thread stack using the --main-stacksize= flag.
>> >>>>>
>> >>>>> ==9947== The main thread stack size used in this run was 10485760.
>> >>>>>
>> >>>>> ==9947==
>> >>>>>
>> >>>>> ==9947== HEAP SUMMARY:
>> >>>>>
>> >>>>> ==9947== in use at exit: 1,181,215,468 bytes in 282,120 blocks
>> >>>>>
>> >>>>> ==9947== total heap usage: 379,556 allocs, 97,436 frees,
>> 2,096,403,341
>> >>>>> bytes allocated
>> >>>>>
>> >>>>> ==9947==
>> >>>>>
>> >>>>> ==9947== LEAK SUMMARY:
>> >>>>>
>> >>>>> ==9947== definitely lost: 516 bytes in 12 blocks
>> >>>>>
>> >>>>> ==9947== indirectly lost: 0 bytes in 0 blocks
>> >>>>>
>> >>>>> ==9947== possibly lost: 33,561,793 bytes in 24 blocks
>> >>>>>
>> >>>>> ==9947== still reachable: 1,147,653,159 bytes in 282,084 blocks
>> >>>>>
>> >>>>> ==9947== suppressed: 0 bytes in 0 blocks
>> >>>>>
>> >>>>> ==9947== Rerun with --leak-check=full to see details of leaked
>> memory
>> >>>>>
>> >>>>> ==9947==
>> >>>>>
>> >>>>> ==9947== For counts of detected and suppressed errors, rerun with:
>> -v
>> >>>>>
>> >>>>> ==9947== Use --track-origins=yes to see where uninitialised values
>> come from
>> >>>>>
>> >>>>> ==9947== ERROR SUMMARY: 25 errors from 5 contexts (suppressed: 100
>> from 9)
>> >>>>>
>> >>>>>
>> >>>>> Seems very similar with before.
>> >>>>>
>> >>>>> BTW: is it the "workerthreads" options make sense to
>> mmjsonparse/mmfields
>> >>>>> etc?
>> >>>>>
>> >>>>> 2015-03-22 0:07 GMT+08:00 Rainer Gerhards <rgerha...@hq.adiscon.com
>> >:
>> >>>>>
>> >>>>>> 2015-03-21 14:50 GMT+01:00 chenlin rao <rao.chen...@gmail.com>:
>> >>>>>> > $MaxMessageSize 32m
>> >>>>>> > module( load="imtcp" )
>> >>>>>> > module( load="imuxsock" )
>> >>>>>> > module( load="imklog" )
>> >>>>>> > module( load="mmfields" )
>> >>>>>> > module( load="mmjsonparse" )
>> >>>>>> > module( load="omelasticsearch" )
>> >>>>>> > $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
>> >>>>>> > $RepeatedMsgReduction off
>> >>>>>> > $WorkDirectory /data0/rsyslog
>> >>>>>> > $MainMsgQueueFilename mainQ
>> >>>>>> > $MainMsgQueueType LinkedList
>> >>>>>> > $MainMsgQueueMaxFileSize 512M
>> >>>>>> > $MainMsgQueueMaxDiskSpace 20G
>> >>>>>> > $PreserveFQDN on
>> >>>>>> > input(type="imtcp" port="514")
>> >>>>>> > template(name="local6IndexName" type="list") {
>> >>>>>> > constant(value="logstash-mweibo-")
>> >>>>>> > property(name="timereported" dateFormat="rfc3339"
>> position.from="1"
>> >>>>>> > position.to="4")
>> >>>>>> > constant(value=".")
>> >>>>>> > property(name="timereported" dateFormat="rfc3339"
>> position.from="6"
>> >>>>>> > position.to="7")
>> >>>>>> > constant(value=".")
>> >>>>>> > property(name="timereported" dateFormat="rfc3339"
>> position.from="9"
>> >>>>>> > position.to="10")
>> >>>>>> > }
>> >>>>>> > template( name="local6TypeName" type="string"
>> string="%programname%" )
>> >>>>>> > template( name="local6JsonArray" type="list" ) {
>> >>>>>> > constant(value="{\"@timestamp\":\"")
>> property(name="timereported"
>> >>>>>> > dateFormat="rfc3339")
>> >>>>>> > constant(value="\",\"host\":\"") property(name="hostname")
>> >>>>>> > constant(value="\",") property(name="$.line"
>> position.from="2")
>> >>>>>> > }
>> >>>>>> > Ruleset( name="forwardRuleSetJsonArray" )
>> >>>>>> > {
>> >>>>>> > action ( type="mmjsonparse"
>> >>>>>> > name="action_jsonarray-parse"
>> >>>>>> > queue.size="3000"
>> >>>>>> > queue.dequeuebatchsize="300"
>> >>>>>> > queue.maxdiskspace="15G"
>> >>>>>> > queue.checkpointinterval="10"
>> >>>>>> > queue.type="linkedlist"
>> >>>>>> > queue.workerthreads="30"
>> >>>>>> > queue.workerthreadminimummessages="100"
>> >>>>>> > queue.maxfilesize="500M"
>> >>>>>> > queue.saveonshutdown="on"
>> >>>>>> > )
>> >>>>>>
>> >>>>>> running mmjsonparse on a queue (asynchronously!) does not make any
>> >>>>>> sense. When running async, the json properties will never be seen
>> by
>> >>>>>> the rest of the ruleset.
>> >>>>>>
>> >>>>>> Maybe this even triggers the fault...
>> >>>>>>
>> >>>>>> Rainer
>> >>>>>> > # maybe recieved a too large data being truncated
>> >>>>>> > if ( re_match($msg, ']}$') ) then {
>> >>>>>> > foreach ($.line in $!msg) do {
>> >>>>>> > action (
>> >>>>>> > # test for other output modules
>> >>>>>> > # type="omfile" file="/data/rsyslog-debug.log"
>> >>>>>> > # type="omfwd" Target="127.0.0.1" Port="5140"
>> >>>>>> Protocol="tcp"
>> >>>>>> > type="omelasticsearch" server="es.domain.com"
>> >>>>>> > dynSearchIndex="on" searchIndex="local6IndexName"
>> dynSearchType="on"
>> >>>>>> > searchType="local6TypeName" asyncrepl="on" bulkmode="on"
>> >>>>>> > template="local6JsonArray"
>> >>>>>> > name="action_json2log-es1003"
>> >>>>>> > queue.filename="action_json2log-es1003"
>> >>>>>> > queue.size="10000"
>> >>>>>> > queue.dequeuebatchsize="2000"
>> >>>>>> > queue.maxdiskspace="15G"
>> >>>>>> > queue.discardseverity="3"
>> >>>>>> > queue.checkpointinterval="10"
>> >>>>>> > queue.type="linkedlist"
>> >>>>>> > queue.workerthreads="5"
>> >>>>>> > queue.workerthreadminimummessages="2000"
>> >>>>>> > queue.maxfilesize="500M"
>> >>>>>> > queue.saveonshutdown="on"
>> >>>>>> > )
>> >>>>>> > }
>> >>>>>> > stop
>> >>>>>> > }
>> >>>>>> > }
>> >>>>>> >
>> >>>>>> > if ( $programname == 'mobile_client_net_fatal_error' ) then
>> >>>>>> > {
>> >>>>>> > call forwardRuleSetJsonArray
>> >>>>>> > stop
>> >>>>>> > }
>> >>>>>> >
>> >>>>>>
>> *.info;mail.none;authpriv.none;cron.none;local6.none;local7.none;user.none
>> >>>>>> > /var/log/messages
>> >>>>>> > authpriv.*
>> /var/log/secure
>> >>>>>> > mail.*
>> /var/log/maillog
>> >>>>>> > cron.*
>> /var/log/cron
>> >>>>>> > uucp,news.crit
>> /var/log/spooler
>> >>>>>> >
>> >>>>>> > 2015-03-21 13:14 GMT+08:00 singh.janmejay <
>> singh.janme...@gmail.com>:
>> >>>>>> >
>> >>>>>> >> Can you please share your config as well?
>> >>>>>> >>
>> >>>>>> >> Also, I'll likely be able to look at it only after 27th Mar. As
>> of now I
>> >>>>>> >> don't have access to a computer and github doesn't seem to show
>> line
>> >>>>>> >> numbers while browsing code using mobile phone.
>> >>>>>> >>
>> >>>>>> >> --
>> >>>>>> >> Regards,
>> >>>>>> >> Janmejay
>> >>>>>> >>
>> >>>>>> >> PS: Please blame the typos in this mail on my phone's
>> uncivilized soft
>> >>>>>> >> keyboard sporting it's not-so-smart-assist technology.
>> >>>>>> >>
>> >>>>>> >> On Mar 21, 2015 8:49 AM, "chenlin rao" <rao.chen...@gmail.com>
>> wrote:
>> >>>>>> >>
>> >>>>>> >> > I try to build rsyslogd from github master with "./configure
>> >>>>>> >> --enable-debug
>> >>>>>> >> > --enable-valgrind --enable-memcheck --enable-elasticsearch
>> >>>>>> >> > --enable-mmjsonparse --enable-mmsequence --enable-mmfields
>> >>>>>> >> > --disable-liblogging-stdlog --enable-omruleset"
>> >>>>>> >> >
>> >>>>>> >> > then process exit with:
>> >>>>>> >> >
>> >>>>>> >> > 7778.301430428:main Q[DA]:Reg/w0: ACTION 16
>> >>>>>> >> > [builtin:omfwd:action(type="builtin:omfwd" ...)]
>> >>>>>> >> > 7778.301676909:main Q[DA]:Reg/w0: executing action 16
>> >>>>>> >> > 7778.301791807:main Q[DA]:Reg/w0: Called action, logging to
>> >>>>>> builtin:omfwd
>> >>>>>> >> > ==20833== Thread 2:
>> >>>>>> >> > ==20833== Invalid read of size 4
>> >>>>>> >> > ==20833== at 0x3A044093A0: pthread_mutex_lock (in /lib64/
>> >>>>>> >> > libpthread-2.12.so)
>> >>>>>> >> > ==20833== by 0x438053: qqueueEnqMsg (queue.c:2856)
>> >>>>>> >> > ==20833== by 0x4421AB: doSubmitToActionQ (action.c:1403)
>> >>>>>> >> > ==20833== by 0x439D6C: scriptExec (ruleset.c:197)
>> >>>>>> >> > ==20833== by 0x439ECC: scriptExec (ruleset.c:284)
>> >>>>>> >> > ==20833== by 0x439CD9: scriptExec (ruleset.c:416)
>> >>>>>> >> > ==20833== by 0x43A40B: processBatch (ruleset.c:503)
>> >>>>>> >> > ==20833== by 0x44A403: msgConsumer (rsyslogd.c:575)
>> >>>>>> >> > ==20833== by 0x438D12: ConsumerReg (queue.c:1897)
>> >>>>>> >> > ==20833== by 0x43314B: wtiWorker (wti.c:334)
>> >>>>>> >> > ==20833== by 0x431BA6: wtpWorker (wtp.c:389)
>> >>>>>> >> > ==20833== by 0x3A044079D0: start_thread (in /lib64/
>> >>>>>> libpthread-2.12.so
>> >>>>>> >> )
>> >>>>>> >> > ==20833== Address 0x10 is not stack'd, malloc'd or
>> (recently) free'd
>> >>>>>> >> > ==20833==
>> >>>>>> >> > ==20833==
>> >>>>>> >> > ==20833== Process terminating with default action of signal 11
>> >>>>>> (SIGSEGV)
>> >>>>>> >> > ==20833== Access not within mapped region at address 0x10
>> >>>>>> >> > ==20833== at 0x3A044093A0: pthread_mutex_lock (in /lib64/
>> >>>>>> >> > libpthread-2.12.so)
>> >>>>>> >> > ==20833== by 0x438053: qqueueEnqMsg (queue.c:2856)
>> >>>>>> >> > ==20833== by 0x4421AB: doSubmitToActionQ (action.c:1403)
>> >>>>>> >> > ==20833== by 0x439D6C: scriptExec (ruleset.c:197)
>> >>>>>> >> > ==20833== by 0x439ECC: scriptExec (ruleset.c:284)
>> >>>>>> >> > ==20833== by 0x439CD9: scriptExec (ruleset.c:416)
>> >>>>>> >> > ==20833== by 0x43A40B: processBatch (ruleset.c:503)
>> >>>>>> >> > ==20833== by 0x44A403: msgConsumer (rsyslogd.c:575)
>> >>>>>> >> > ==20833== by 0x438D12: ConsumerReg (queue.c:1897)
>> >>>>>> >> > ==20833== by 0x43314B: wtiWorker (wti.c:334)
>> >>>>>> >> > ==20833== by 0x431BA6: wtpWorker (wtp.c:389)
>> >>>>>> >> > ==20833== by 0x3A044079D0: start_thread (in /lib64/
>> >>>>>> libpthread-2.12.so
>> >>>>>> >> )
>> >>>>>> >> > ==20833== If you believe this happened as a result of a stack
>> >>>>>> >> > ==20833== overflow in your program's main thread (unlikely
>> but
>> >>>>>> >> > ==20833== possible), you can try to increase the size of the
>> >>>>>> >> > ==20833== main thread stack using the --main-stacksize= flag.
>> >>>>>> >> > ==20833== The main thread stack size used in this run was
>> 10485760.
>> >>>>>> >> > ==20833==
>> >>>>>> >> > ==20833== HEAP SUMMARY:
>> >>>>>> >> > ==20833== in use at exit: 44,605,326 bytes in 5,019 blocks
>> >>>>>> >> > ==20833== total heap usage: 10,310 allocs, 5,291 frees,
>> 78,621,963
>> >>>>>> >> bytes
>> >>>>>> >> > allocated
>> >>>>>> >> > ==20833==
>> >>>>>> >> > ==20833== LEAK SUMMARY:
>> >>>>>> >> > ==20833== definitely lost: 516 bytes in 12 blocks
>> >>>>>> >> > ==20833== indirectly lost: 0 bytes in 0 blocks
>> >>>>>> >> > ==20833== possibly lost: 33,559,553 bytes in 17 blocks
>> >>>>>> >> > ==20833== still reachable: 11,045,257 bytes in 4,990 blocks
>> >>>>>> >> > ==20833== suppressed: 0 bytes in 0 blocks
>> >>>>>> >> > ==20833== Rerun with --leak-check=full to see details of
>> leaked memory
>> >>>>>> >> > ==20833==
>> >>>>>> >> > ==20833== For counts of detected and suppressed errors, rerun
>> with: -v
>> >>>>>> >> > ==20833== Use --track-origins=yes to see where uninitialised
>> values
>> >>>>>> come
>> >>>>>> >> > from
>> >>>>>> >> > ==20833== ERROR SUMMARY: 11 errors from 3 contexts
>> (suppressed: 100
>> >>>>>> from
>> >>>>>> >> 9)
>> >>>>>> >> > 已杀死
>> >>>>>> >> >
>> >>>>>> >> > 2015-03-20 23:29 GMT+08:00 singh.janmejay <
>> singh.janme...@gmail.com>:
>> >>>>>> >> >
>> >>>>>> >> > > Can you please build with debug symbols and repeat the
>> valgrind run?
>> >>>>>> >> > >
>> >>>>>> >> > > --
>> >>>>>> >> > > Regards,
>> >>>>>> >> > > Janmejay
>> >>>>>> >> > >
>> >>>>>> >> > > PS: Please blame the typos in this mail on my phone's
>> uncivilized
>> >>>>>> soft
>> >>>>>> >> > > keyboard sporting it's not-so-smart-assist technology.
>> >>>>>> >> > >
>> >>>>>> >> > > On Mar 20, 2015 6:03 PM, "chenlin rao" <
>> rao.chen...@gmail.com>
>> >>>>>> wrote:
>> >>>>>> >> > >
>> >>>>>> >> > > > btw: if I change omelasticsearch/omfwd to omfile,
>> rsyslogd would
>> >>>>>> be
>> >>>>>> >> > > fine...
>> >>>>>> >> > > >
>> >>>>>> >> > > > 2015-03-20 20:13 GMT+08:00 chenlin rao <
>> rao.chen...@gmail.com>:
>> >>>>>> >> > > >
>> >>>>>> >> > > > > 3498.767218405:main Q[DA]:Reg/w0: rainerscript: var
>> 200:!msg:
>> >>>>>> '[ {
>> >>>>>> >> > > "uid":
>> >>>>>> >> > > > > "1941604034", "request_header":
>> >>>>>> >> > > "{\"Accept-Encoding\":\"gzip,deflate\"}",
>> >>>>>> >> > > > > "network_type": "wifi", "end_time": "1426836307406",
>> "dns_ip":
>> >>>>>> >> > > > > "218.15.203.34,192.168.1.1", "response_code": "200",
>> >>>>>> >> "response_data":
>> >>>>>> >> > > > "{}",
>> >>>>>> >> > > > > "start_time": "1426836307328", "act":
>> "net_fatal_error", "type":
>> >>>>>> >> > > > > "net_fatal_error", "request_url": "https:\/\/
>> api.weibo.cn
>> >>>>>> >> > > >
>> >>>>>> >> > >
>> >>>>>> >> >
>> >>>>>> >>
>> >>>>>>
>> \/2\/client\/url_list?accuracy_level=0&c=android&i=9eef7ba&s=0088f6a4&ua=Xiaomi-MI%202S__weibo__5.0.0__android__android4.4.4&wm=5311_4002&v_f=2&from=1050095010&gsid=4uzH00573m6YZU5oVhYra896c0y&lang=zh_CN&skin=default&oldwm=5311_4002",
>> >>>>>> >> > > > > "response_status_line": "HTTP\/1.1 200 OK" }, { "uid":
>> >>>>>> >> "1941604034",
>> >>>>>> >> > > > > "request_header":
>> "{\"Accept-Encoding\":\"gzip,deflate\"}",
>> >>>>>> >> > > > "network_type":
>> >>>>>> >> > > > > "wifi", "end_time": "1426836320563", "dns_ip":
>> >>>>>> >> > > > "218.15.203.34,192.168.1.1",
>> >>>>>> >> > > > > "response_code": "200", "response_data": "{}",
>> "start_time":
>> >>>>>> >> > > > > "1426836320505", "act": "net_fatal_error", "type":
>> >>>>>> >> "net_fatal_error",
>> >>>>>> >> > > > > "request_url": "https:\/\/api.weibo.cn
>> >>>>>> >> > > >
>> >>>>>> >> > >
>> >>>>>> >> >
>> >>>>>> >>
>> >>>>>>
>> \/2\/client\/url_list?accuracy_level=0&c=android&i=9eef7ba&s=0088f6a4&ua=Xiaomi-MI%202S__weibo__5.0.0__android__android4.4.4&wm=5311_4002&v_f=2&from=1050095010&gsid=4uzH00573m6YZU5oVhYra896c0y&lang=zh_CN&skin=default&oldwm=5311_4002",
>> >>>>>> >> > > > > "response_status_line": "HTTP\/1.1 200 OK" } ]'
>> >>>>>> >> > > > > 3498.767338611:main Q[DA]:Reg/w0: eval expr 0x62e63a0,
>> return
>> >>>>>> >> > datatype
>> >>>>>> >> > > > 'J'
>> >>>>>> >> > > > > 3498.770732236:main Q[DA]:Reg/w0: ACTION 17
>> >>>>>> >> > > > > [builtin:omfwd:action(type="builtin:omfwd" ...)]
>> >>>>>> >> > > > > 3498.770963455:main Q[DA]:Reg/w0: executing action 17
>> >>>>>> >> > > > > 3498.771052511:main Q[DA]:Reg/w0: Called action,
>> logging to
>> >>>>>> >> > > builtin:omfwd
>> >>>>>> >> > > > > ==30138== Thread 3:
>> >>>>>> >> > > > > ==30138== Invalid read of size 4
>> >>>>>> >> > > > > ==30138== at 0x4E3E3A0: pthread_mutex_lock (in
>> /lib64/
>> >>>>>> >> > > > > libpthread-2.12.so)
>> >>>>>> >> > > > > ==30138== by 0x14228D: qqueueEnqMsg (in
>> /sbin/rsyslogd)
>> >>>>>> >> > > > > ==30138== by 0x14BB5B: ??? (in /sbin/rsyslogd)
>> >>>>>> >> > > > > ==30138== by 0x143C4C: ??? (in /sbin/rsyslogd)
>> >>>>>> >> > > > > ==30138== by 0x143DAC: ??? (in /sbin/rsyslogd)
>> >>>>>> >> > > > > ==30138== by 0x143BB9: ??? (in /sbin/rsyslogd)
>> >>>>>> >> > > > > ==30138== by 0x1442CB: ??? (in /sbin/rsyslogd)
>> >>>>>> >> > > > > ==30138== by 0x15336B: ??? (in /sbin/rsyslogd)
>> >>>>>> >> > > > > ==30138== by 0x143112: ??? (in /sbin/rsyslogd)
>> >>>>>> >> > > > > ==30138== by 0x13D671: wtiWorker (in /sbin/rsyslogd)
>> >>>>>> >> > > > > ==30138== by 0x13D1C1: ??? (in /sbin/rsyslogd)
>> >>>>>> >> > > > > ==30138== by 0x4E3C9D0: start_thread (in /lib64/
>> >>>>>> >> > libpthread-2.12.so)
>> >>>>>> >> > > > > ==30138== Address 0x10 is not stack'd, malloc'd or
>> (recently)
>> >>>>>> >> free'd
>> >>>>>> >> > > > > ==30138==
>> >>>>>> >> > > > > ==30138==
>> >>>>>> >> > > > > ==30138== Process terminating with default action of
>> signal 11
>> >>>>>> >> > > (SIGSEGV)
>> >>>>>> >> > > > > ==30138== Access not within mapped region at address
>> 0x10
>> >>>>>> >> > > > > ==30138== at 0x4E3E3A0: pthread_mutex_lock (in
>> /lib64/
>> >>>>>> >> > > > > libpthread-2.12.so)
>> >>>>>> >> > > > > ==30138== by 0x14228D: qqueueEnqMsg (in
>> /sbin/rsyslogd)
>> >>>>>> >> > > > > ==30138== by 0x14BB5B: ??? (in /sbin/rsyslogd)
>> >>>>>> >> > > > > ==30138== by 0x143C4C: ??? (in /sbin/rsyslogd)
>> >>>>>> >> > > > > ==30138== by 0x143DAC: ??? (in /sbin/rsyslogd)
>> >>>>>> >> > > > > ==30138== by 0x143BB9: ??? (in /sbin/rsyslogd)
>> >>>>>> >> > > > > ==30138== by 0x1442CB: ??? (in /sbin/rsyslogd)
>> >>>>>> >> > > > > ==30138== by 0x15336B: ??? (in /sbin/rsyslogd)
>> >>>>>> >> > > > > ==30138== by 0x143112: ??? (in /sbin/rsyslogd)
>> >>>>>> >> > > > > ==30138== by 0x13D671: wtiWorker (in /sbin/rsyslogd)
>> >>>>>> >> > > > > ==30138== by 0x13D1C1: ??? (in /sbin/rsyslogd)
>> >>>>>> >> > > > > ==30138== by 0x4E3C9D0: start_thread (in /lib64/
>> >>>>>> >> > libpthread-2.12.so)
>> >>>>>> >> > > > > ==30138== If you believe this happened as a result of
>> a stack
>> >>>>>> >> > > > > ==30138== overflow in your program's main thread
>> (unlikely but
>> >>>>>> >> > > > > ==30138== possible), you can try to increase the size
>> of the
>> >>>>>> >> > > > > ==30138== main thread stack using the
>> --main-stacksize= flag.
>> >>>>>> >> > > > > ==30138== The main thread stack size used in this run
>> was
>> >>>>>> >> 10485760.
>> >>>>>> >> > > > > ==30138==
>> >>>>>> >> > > > > ==30138== HEAP SUMMARY:
>> >>>>>> >> > > > > ==30138== in use at exit: 1,078,127,378 bytes in
>> 291,997
>> >>>>>> blocks
>> >>>>>> >> > > > > ==30138== total heap usage: 614,872 allocs, 322,875
>> frees,
>> >>>>>> >> > > > 1,586,744,205
>> >>>>>> >> > > > > bytes allocated
>> >>>>>> >> > > > > ==30138==
>> >>>>>> >> > > > > ==30138== LEAK SUMMARY:
>> >>>>>> >> > > > > ==30138== definitely lost: 33,555,027 bytes in 13
>> blocks
>> >>>>>> >> > > > > ==30138== indirectly lost: 0 bytes in 0 blocks
>> >>>>>> >> > > > > ==30138== possibly lost: 5,760 bytes in 18 blocks
>> >>>>>> >> > > > > ==30138== still reachable: 1,044,566,591 bytes in
>> 291,966
>> >>>>>> blocks
>> >>>>>> >> > > > > ==30138== suppressed: 0 bytes in 0 blocks
>> >>>>>> >> > > > > ==30138== Rerun with --leak-check=full to see details
>> of leaked
>> >>>>>> >> > memory
>> >>>>>> >> > > > > ==30138==
>> >>>>>> >> > > > > ==30138== For counts of detected and suppressed errors,
>> rerun
>> >>>>>> with:
>> >>>>>> >> > -v
>> >>>>>> >> > > > > ==30138== Use --track-origins=yes to see where
>> uninitialised
>> >>>>>> values
>> >>>>>> >> > > come
>> >>>>>> >> > > > > from
>> >>>>>> >> > > > > ==30138== ERROR SUMMARY: 1809 errors from 5 contexts
>> >>>>>> (suppressed:
>> >>>>>> >> 131
>> >>>>>> >> > > > from
>> >>>>>> >> > > > > 9)
>> >>>>>> >> > > > > Killed
>> >>>>>> >> > > > >
>> >>>>>> >> > > > >
>> >>>>>> >> > > > > 2015-03-20 8:50 GMT+08:00 singh.janmejay <
>> >>>>>> singh.janme...@gmail.com
>> >>>>>> >> >:
>> >>>>>> >> > > > >
>> >>>>>> >> > > > >> Can you please run this with valgrind and share its
>> output on
>> >>>>>> >> crash?
>> >>>>>> >> > > > >>
>> >>>>>> >> > > > >> --
>> >>>>>> >> > > > >> Regards,
>> >>>>>> >> > > > >> Janmejay
>> >>>>>> >> > > > >>
>> >>>>>> >> > > > >> PS: Please blame the typos in this mail on my phone's
>> >>>>>> uncivilized
>> >>>>>> >> > soft
>> >>>>>> >> > > > >> keyboard sporting it's not-so-smart-assist technology.
>> >>>>>> >> > > > >>
>> >>>>>> >> > > > >> On Mar 19, 2015 11:10 PM, "chenlin rao" <
>> rao.chen...@gmail.com
>> >>>>>> >
>> >>>>>> >> > > wrote:
>> >>>>>> >> > > > >>
>> >>>>>> >> > > > >> > Hello everyone.
>> >>>>>> >> > > > >> >
>> >>>>>> >> > > > >> > I just learnt a foreach syntax from
>> >>>>>> >> > > `src/tests/json_array_looping.sh`,
>> >>>>>> >> > > > >> so I
>> >>>>>> >> > > > >> > try to parse my logdata(yes, long json array in msg)
>> as
>> >>>>>> follow:
>> >>>>>> >> > > > >> >
>> >>>>>> >> > > > >> > ```
>> >>>>>> >> > > > >> >
>> >>>>>> >> > > > >> > $MaxMessageSize 256k
>> >>>>>> >> > > > >> >
>> >>>>>> >> > > > >> > template( name="local6JsonArray" type="string"
>> >>>>>> >> > string="%$.line%\n" )
>> >>>>>> >> > > > >> >
>> >>>>>> >> > > > >> > Ruleset( name="forwardRuleSetJsonArray" )
>> >>>>>> >> > > > >> >
>> >>>>>> >> > > > >> > {
>> >>>>>> >> > > > >> >
>> >>>>>> >> > > > >> > action( type="mmjsonparse" )
>> >>>>>> >> > > > >> >
>> >>>>>> >> > > > >> > foreach ($.line in $!msg) do {
>> >>>>>> >> > > > >> >
>> >>>>>> >> > > > >> > action ( type="omfile" file="/data0/logfile"
>> >>>>>> >> > > > >> > template="local6JsonArray")
>> >>>>>> >> > > > >> > }
>> >>>>>> >> > > > >> > }
>> >>>>>> >> > > > >> > ```
>> >>>>>> >> > > > >> >
>> >>>>>> >> > > > >> > But I always got segment fault after restarted few
>> minutes.
>> >>>>>> >> > > > >> >
>> >>>>>> >> > > > >> > Before the rsyslogd died , I can watch some correct
>> lines in
>> >>>>>> the
>> >>>>>> >> > > > >> > "/data0/logfile".
>> >>>>>> >> > > > >> >
>> >>>>>> >> > > > >> > While running `rsyslogd -dn`, I watch the last five
>> lines
>> >>>>>> were:
>> >>>>>> >> > > > >> >
>> >>>>>> >> > > > >> > ```
>> >>>>>> >> > > > >> >
>> >>>>>> >> > > > >> > 3295.820218609:main Q:Reg/w0 : FOREACH .line IN
>> >>>>>> >> > > > >> >
>> >>>>>> >> > > > >> > 3295.820227632:main Q:Reg/w0 : var '!msg'
>> >>>>>> >> > > > >> >
>> >>>>>> >> > > > >> > 3295.820240109:main Q:Reg/w0 : eval expr
>> 0x7fe2044fefa0,
>> >>>>>> type
>> >>>>>> >> > > 'V[86]'
>> >>>>>> >> > > > >> >
>> >>>>>> >> > > > >> > 3295.821747011:main Q:Reg/w0 : rainerscript: var
>> 200:!msg:
>> >>>>>> >> > > '{"msg":[{
>> >>>>>> >> > > > >> > "end_time":"142...longmsghere....\"verieval expr
>> >>>>>> 0x7fe2044fefa0,
>> >>>>>> >> > > > return
>> >>>>>> >> > > > >> > datatype 'J'
>> >>>>>> >> > > > >> > ```
>> >>>>>> >> > > > >> >
>> >>>>>> >> > > > >> > The length of such line is about 20k~30k, far away
>> below my
>> >>>>>> >> > > > >> > $MaxMessageSize.
>> >>>>>> >> > > > >> >
>> >>>>>> >> > > > >> > So, why rsyslogd segment fault?
>> >>>>>> >> > > > >> > _______________________________________________
>> >>>>>> >> > > > >> > rsyslog mailing list
>> >>>>>> >> > > > >> > http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >>>>>> >> > > > >> > http://www.rsyslog.com/professional-services/
>> >>>>>> >> > > > >> > What's up with rsyslog? Follow
>> https://twitter.com/rgerhards
>> >>>>>> >> > > > >> > NOTE WELL: This is a PUBLIC mailing list, posts are
>> ARCHIVED
>> >>>>>> by
>> >>>>>> >> a
>> >>>>>> >> > > > myriad
>> >>>>>> >> > > > >> > of sites beyond our control. PLEASE UNSUBSCRIBE and
>> DO NOT
>> >>>>>> POST
>> >>>>>> >> if
>> >>>>>> >> > > you
>> >>>>>> >> > > > >> > DON'T LIKE THAT.
>> >>>>>> >> > > > >> >
>> >>>>>> >> > > > >> _______________________________________________
>> >>>>>> >> > > > >> rsyslog mailing list
>> >>>>>> >> > > > >> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >>>>>> >> > > > >> http://www.rsyslog.com/professional-services/
>> >>>>>> >> > > > >> What's up with rsyslog? Follow
>> https://twitter.com/rgerhards
>> >>>>>> >> > > > >> NOTE WELL: This is a PUBLIC mailing list, posts are
>> ARCHIVED
>> >>>>>> by a
>> >>>>>> >> > > myriad
>> >>>>>> >> > > > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO
>> NOT
>> >>>>>> POST if
>> >>>>>> >> > you
>> >>>>>> >> > > > >> DON'T LIKE THAT.
>> >>>>>> >> > > > >>
>> >>>>>> >> > > > >
>> >>>>>> >> > > > >
>> >>>>>> >> > > > _______________________________________________
>> >>>>>> >> > > > rsyslog mailing list
>> >>>>>> >> > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >>>>>> >> > > > http://www.rsyslog.com/professional-services/
>> >>>>>> >> > > > What's up with rsyslog? Follow
>> https://twitter.com/rgerhards
>> >>>>>> >> > > > NOTE WELL: This is a PUBLIC mailing list, posts are
>> ARCHIVED by a
>> >>>>>> >> > myriad
>> >>>>>> >> > > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO
>> NOT POST if
>> >>>>>> >> you
>> >>>>>> >> > > > DON'T LIKE THAT.
>> >>>>>> >> > > >
>> >>>>>> >> > > _______________________________________________
>> >>>>>> >> > > rsyslog mailing list
>> >>>>>> >> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >>>>>> >> > > http://www.rsyslog.com/professional-services/
>> >>>>>> >> > > What's up with rsyslog? Follow
>> https://twitter.com/rgerhards
>> >>>>>> >> > > NOTE WELL: This is a PUBLIC mailing list, posts are
>> ARCHIVED by a
>> >>>>>> >> myriad
>> >>>>>> >> > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>> POST if
>> >>>>>> you
>> >>>>>> >> > > DON'T LIKE THAT.
>> >>>>>> >> > >
>> >>>>>> >> > _______________________________________________
>> >>>>>> >> > rsyslog mailing list
>> >>>>>> >> > http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >>>>>> >> > http://www.rsyslog.com/professional-services/
>> >>>>>> >> > What's up with rsyslog? Follow https://twitter.com/rgerhards
>> >>>>>> >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
>> by a
>> >>>>>> myriad
>> >>>>>> >> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>> POST if you
>> >>>>>> >> > DON'T LIKE THAT.
>> >>>>>> >> _______________________________________________
>> >>>>>> >> rsyslog mailing list
>> >>>>>> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >>>>>> >> http://www.rsyslog.com/professional-services/
>> >>>>>> >> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> >>>>>> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by
>> a myriad
>> >>>>>> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
>> if you
>> >>>>>> >> DON'T LIKE THAT.
>> >>>>>> >>
>> >>>>>> > _______________________________________________
>> >>>>>> > rsyslog mailing list
>> >>>>>> > http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >>>>>> > http://www.rsyslog.com/professional-services/
>> >>>>>> > What's up with rsyslog? Follow https://twitter.com/rgerhards
>> >>>>>> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by
>> a myriad
>> >>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
>> you
>> >>>>>> DON'T LIKE THAT.
>> >>>>>> _______________________________________________
>> >>>>>> rsyslog mailing list
>> >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >>>>>> http://www.rsyslog.com/professional-services/
>> >>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> myriad
>> >>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
>> you
>> >>>>>> DON'T LIKE THAT.
>> >>>>>>
>> >>>>> _______________________________________________
>> >>>>> rsyslog mailing list
>> >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >>>>> http://www.rsyslog.com/professional-services/
>> >>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
>> you DON'T LIKE THAT.
>> >>>>
>> >>>> _______________________________________________
>> >>>> rsyslog mailing list
>> >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >>>> http://www.rsyslog.com/professional-services/
>> >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
>> you DON'T LIKE THAT.
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Regards,
>> >>> Janmejay
>> >>> http://codehunk.wordpress.com
>> >>> _______________________________________________
>> >>> rsyslog mailing list
>> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >>> http://www.rsyslog.com/professional-services/
>> >>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
>> you DON'T LIKE THAT.
>> >> _______________________________________________
>> >> rsyslog mailing list
>> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >> http://www.rsyslog.com/professional-services/
>> >> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
>> you DON'T LIKE THAT.
>> >
>> >
>> >
>> > --
>> > Regards,
>> > Janmejay
>> > http://codehunk.wordpress.com
>>
>>
>>
>> --
>> Regards,
>> Janmejay
>> http://codehunk.wordpress.com
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>>
>
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
