Hello, We have observed several coredumps of rsyslogd (imjournal plugin), receiving signal SIGSEGV. The reason appears to be that that sd_journal_get_data() returns <0 when trying to retrieve the MESSAGE field. As I saw it, the flow was:
Starting at imjournal.c:247 : if (sd_journal_get_data(j, "MESSAGE", &get,
&length) < 0)
sd_journal_get_data() returned <0. Then the string message is set to "".
Then enqMsg() was called, with message=""
After that, SanitizeMsg() was called, with
pMsg->pszRawMsg = ""
pMsg->iLenRawMsg = 0,
which violates the assertions at parser.c:404:
assert(pMsg != NULL);
assert(pMsg->iLenRawMsg > 0);
Then rsyslogd received a SIGSEGV at parser.c:483.
Maybe the handling of the case when sd_journal_get_data() returns <0 is not the
best one, although I do not know the code well in order to suggest something
better.
Any help is greatly appreciated.
Attached is the backtrace from the coredump, plus the rsyslog configuration
file.
We use version 8.10.0:
# /usr/sbin/rsyslogd -version
rsyslogd 8.10.0, compiled with:
PLATFORM: x86_64-unknown-linux-gnu
PLATFORM (lsb_release -d):
FEATURE_REGEXP: Yes
GSSAPI Kerberos 5 support: No
FEATURE_DEBUG (debug build, slow code): No
32bit Atomic operations supported: Yes
64bit Atomic operations supported: Yes
memory allocator: system default
Runtime Instrumentation (slow code): No
uuid support: Yes
Number of Bits in RainerScript integers: 64
# systemctl --version
systemd 225
+PAM +AUDIT -SELINUX -IMA -APPARMOR -SMACK +SYSVINIT +UTMP -LIBCRYPTSETUP
-GCRYPT -GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN
Kernel version is 3.18.13.
Best regards,
Klearchos
GNU gdb (GDB) 7.8 Copyright (C) 2014 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/sbin/rsyslogd...Reading symbols from /usr/lib/debug//usr/sbin/rsyslogd.debug...done. done. [New LWP 2452] [New LWP 2846] [New LWP 2453] [New LWP 2923] [New LWP 2449] warning: Could not load shared library symbols for linux-vdso.so.1. Do you need "set solib-search-path" or "set sysroot"? [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `/usr/sbin/rsyslogd -f /etc/rsyslog-local.conf -i /var/run/rsyslog.pid'. Program terminated with signal SIGSEGV, Segmentation fault. #0 SanitizeMsg (pMsg=0x7f6933d0ab10) at parser.c:483 483 parser.c: No such file or directory. (gdb) bt #0 SanitizeMsg (pMsg=0x7f6933d0ab10) at parser.c:483 #1 0x00007f695f7e5030 in enqMsg (json=0x7f6933d06380, tp=0x7f69369d8c50, iSeverity=5, iFacility=3, pszTag=0x7f6933d08cf0 "journal:", msg=0x7f6933d025a0 "") at imjournal.c:193 #2 readjournal () at imjournal.c:415 #3 runInput (pThrd=<optimized out>) at imjournal.c:634 #4 0x000000000043c33e in thrdStarter (arg=0x18d54b0) at ../threads.c:212 #5 0x00007f6962201324 in start_thread (arg=0x7f69369d9700) at pthread_create.c:333 #6 0x00007f696132297d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 (gdb) thread apply all bt Thread 5 (Thread 0x7f696283a800 (LWP 2449)): #0 0x00007f696131b8b3 in select () at ../sysdeps/unix/syscall-template.S:81 #1 0x000000000040da49 in mainloop () at rsyslogd.c:1494 #2 main (argc=<optimized out>, argv=0x7ffc62ecb328) at rsyslogd.c:1628 Thread 4 (Thread 0x7f69547d3700 (LWP 2923)): #0 0x00007f6962209aed in connect () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f69547d5c0b in Connect (pNsd=0x7f6920006620, family=<optimized out>, port=<optimized out>, host=<optimized out>) at nsd_ptcp.c:736 #2 0x000000000044809f in TCPSendInit (pvData=pvData@entry=0x7f69200022a0) at omfwd.c:692 #3 0x00000000004480ed in doTryResume (pWrkrData=0x7f69200022a0) at omfwd.c:741 #4 0x000000000043a99a in actionPrepare (pWti=0x18d22f0, pThis=0x18d01a0) at ../action.c:856 #5 processMsgMain (pAction=pAction@entry=0x18d01a0, pWti=pWti@entry=0x18d22f0, pMsg=<optimized out>, ttNow=ttNow@entry=0x7f69547d2bc0) at ../action.c:1305 #6 0x000000000043ab76 in processBatchMain (pVoid=0x18d01a0, pBatch=0x18d2320, pWti=0x18d22f0) at ../action.c:1342 #7 0x0000000000430923 in ConsumerReg (pThis=0x18d0580, pWti=0x18d22f0) at queue.c:1897 #8 0x000000000042d37e in wtiWorker (pThis=pThis@entry=0x18d22f0) at wti.c:334 #9 0x000000000042c3e4 in wtpWorker (arg=0x18d22f0) at wtp.c:389 #10 0x00007f6962201324 in start_thread (arg=0x7f69547d3700) at pthread_create.c:333 #11 0x00007f696132297d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 3 (Thread 0x7f69361d8700 (LWP 2453)): #0 0x00007f6962209aed in connect () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f69547d5c0b in Connect (pNsd=0x7f69280070a0, family=<optimized out>, port=<optimized out>, host=<optimized out>) at nsd_ptcp.c:736 #2 0x000000000044809f in TCPSendInit (pvData=pvData@entry=0x7f69280023d0) at omfwd.c:692 #3 0x00000000004480ed in doTryResume (pWrkrData=0x7f69280023d0) at omfwd.c:741 #4 0x00000000004386f1 in actionDoRetry (pThis=pThis@entry=0x18d0a50, pWti=pWti@entry=0x18d3b20) at ../action.c:718 #5 0x0000000000439da4 in actionTryResume (pWti=0x18d3b20, pThis=0x18d0a50) at ../action.c:827 #6 actionPrepare (pWti=0x18d3b20, pThis=0x18d0a50) at ../action.c:850 #7 actionTryCommit (pWti=0x18d3b20, pThis=0x18d0a50) at ../action.c:1175 #8 actionCommit (pThis=pThis@entry=0x18d0a50, pWti=pWti@entry=0x18d3b20) at ../action.c:1243 #9 0x000000000043add8 in actionCommitAllDirect (pWti=pWti@entry=0x18d3b20) at ../action.c:1278 #10 0x0000000000434776 in processBatch (pBatch=0x18d3b50, pWti=0x18d3b20) at ruleset.c:517 #11 0x0000000000442b9a in msgConsumer (notNeeded=<optimized out>, pBatch=0x18d3b50, pWti=0x18d3b20) at rsyslogd.c:575 #12 0x0000000000430923 in ConsumerReg (pThis=0x18d32f0, pWti=0x18d3b20) at queue.c:1897 #13 0x000000000042d37e in wtiWorker (pThis=pThis@entry=0x18d3b20) at wti.c:334 #14 0x000000000042c3e4 in wtpWorker (arg=0x18d3b20) at wtp.c:389 #15 0x00007f6962201324 in start_thread (arg=0x7f69361d8700) at pthread_create.c:333 #16 0x00007f696132297d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 2 (Thread 0x7f69357d1700 (LWP 2846)): #0 pthread_cond_timedwait () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 #1 0x00000000004317a3 in doEnqSingleObj (pMsg=0x7f69308c8230, flowCtlType=eFLOWCTL_NO_DELAY, pThis=0x18d0580) at queue.c:2761 #2 qqueueEnqMsg (pThis=0x18d0580, flowCtlType=flowCtlType@entry=eFLOWCTL_NO_DELAY, pMsg=<optimized out>) at queue.c:2859 ---Type <return> to continue, or q <return> to quit--- #3 0x000000000043ac02 in doSubmitToActionQ (pAction=0x18d01a0, pWti=0x18d4670, pMsg=<optimized out>) at ../action.c:1453 #4 0x0000000000433fb5 in execAct (stmt=0x18ca7d0, pWti=0x18d4670, pMsg=0x7f69308c8230) at ruleset.c:202 #5 scriptExec (root=<optimized out>, pMsg=pMsg@entry=0x7f69308c8230, pWti=pWti@entry=0x18d4670) at ruleset.c:454 #6 0x000000000043404b in execPRIFILT (pWti=<optimized out>, pMsg=<optimized out>, stmt=<optimized out>) at ruleset.c:314 #7 scriptExec (root=<optimized out>, pMsg=pMsg@entry=0x7f69308c8230, pWti=pWti@entry=0x18d4670) at ruleset.c:472 #8 0x00000000004346f4 in processBatch (pBatch=0x18d46a0, pWti=0x18d4670) at ruleset.c:508 #9 0x0000000000442b9a in msgConsumer (notNeeded=<optimized out>, pBatch=0x18d46a0, pWti=0x18d4670) at rsyslogd.c:575 #10 0x0000000000430923 in ConsumerReg (pThis=0x18d32f0, pWti=0x18d4670) at queue.c:1897 #11 0x000000000042d37e in wtiWorker (pThis=pThis@entry=0x18d4670) at wti.c:334 #12 0x000000000042c3e4 in wtpWorker (arg=0x18d4670) at wtp.c:389 #13 0x00007f6962201324 in start_thread (arg=0x7f69357d1700) at pthread_create.c:333 #14 0x00007f696132297d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 1 (Thread 0x7f69369d9700 (LWP 2452)): #0 SanitizeMsg (pMsg=0x7f6933d0ab10) at parser.c:483 #1 0x00007f695f7e5030 in enqMsg (json=0x7f6933d06380, tp=0x7f69369d8c50, iSeverity=5, iFacility=3, pszTag=0x7f6933d08cf0 "journal:", msg=0x7f6933d025a0 "") at imjournal.c:193 #2 readjournal () at imjournal.c:415 #3 runInput (pThrd=<optimized out>) at imjournal.c:634 #4 0x000000000043c33e in thrdStarter (arg=0x18d54b0) at ../threads.c:212 #5 0x00007f6962201324 in start_thread (arg=0x7f69369d9700) at pthread_create.c:333 #6 0x00007f696132297d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
rsyslog-local.conf
Description: rsyslog-local.conf
_______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
