I today tried quite a while to reproduce the issue with your config, unfortunately without any success. For now, I conclude that this is not a general problem, but one induced by your environment and data.
I have then looked again at the traces and the few line numbers it contains. Now comes the interesting fact. I notice that there is a mismatch between the line numbers shown and the code that actually was used to build the packages (judging from git and the source RPMs). So I wonder what might have gone wrong here. Can you please let me know which exact repro you installed from and potentially which other repos you have configured on that machine. It might also be useful to query the system from where exactly the packages have ben installed. I found [1], which hopefully helps to get that information. Once I have this information, I can check if there are any version quirks (note that commit [2] fixed a situation that could cause exactly what you see). If there is no issue with the package sources, I unfortunately need your help in further troubleshooting this in your environment. But let's first see what the repo check will bring up. Rainer [1] http://serverfault.com/questions/62026/how-to-know-from-which-yum-repository-a-package-has-been-installed [2] https://github.com/rsyslog/libfastjson/commit/0499ec8df09c913156ead426c171031b566dbe7d 2016-07-23 19:28 GMT+02:00 Alec Swan <[email protected]>: > Yes, that should be OK. Is there a link to the build server/CI env I can > use? > > Thanks, > > Alec > > On Sat, Jul 23, 2016 at 7:08 AM, Rainer Gerhards <[email protected]> > wrote: > >> Thanks. Would it be OK for your if a somewhat mangled version of these >> configs show up inside the testbench? I think about adding a specific >> test for it, that would be great for the future as well. I'd probably >> try to strip it a bit down (e.g. no ES but writing to file, which >> would enable runs on all CI environments and when no ES is available >> -- I assume this would also trigger the bug). >> >> Please let me know, >> Rainer >> >> 2016-07-22 17:46 GMT+02:00 Alec Swan <[email protected]>: >> > Rainer, I attached /etc/rsyslog.conf and >> /etc/rsyslog.d/rsyslog-custom.conf >> > >> > Thanks, >> > >> > Alec >> > >> > On Fri, Jul 22, 2016 at 12:42 AM, Rainer Gerhards < >> [email protected]> >> > wrote: >> > >> >> unfortunately, we do not seem to have debug symbols, so I don't see >> >> much more. Can you share your full config (including all includes, if >> >> any)? Maybe I can get it to abort in my lab as well. >> >> >> >> Rainer >> >> >> >> 2016-07-21 22:02 GMT+02:00 Alec Swan <[email protected]>: >> >> > Rainer, here is the output from running rsyslog under valgrind: >> >> > >> >> > [root valgrind-3.11.0]# valgrind rsyslogd >> >> > ==2371== Memcheck, a memory error detector >> >> > ==2371== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et >> al. >> >> > ==2371== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright >> >> info >> >> > ==2371== Command: rsyslogd >> >> > ==2371== >> >> > ==2371== Conditional jump or move depends on uninitialised value(s) >> >> > ==2371== at 0x16A414: cnfexprDestruct (in /sbin/rsyslogd) >> >> > ==2371== by 0x16B36D: cnfstmtOptimize (in /sbin/rsyslogd) >> >> > ==2371== by 0x146317: ??? (in /sbin/rsyslogd) >> >> > ==2371== by 0x12E5BE: llExecFunc (in /sbin/rsyslogd) >> >> > ==2371== by 0x146176: rulesetOptimizeAll (in /sbin/rsyslogd) >> >> > ==2371== by 0x123EF6: ??? (in /sbin/rsyslogd) >> >> > ==2371== by 0x158A68: ??? (in /sbin/rsyslogd) >> >> > ==2371== by 0x159008: main (in /sbin/rsyslogd) >> >> > ==2371== >> >> > ==2375== Warning: invalid file descriptor 99990 in syscall close() >> >> > ==2371== >> >> > ==2371== HEAP SUMMARY: >> >> > ==2371== in use at exit: 209,122 bytes in 2,121 blocks >> >> > ==2371== total heap usage: 11,004 allocs, 8,883 frees, 4,319,525 >> bytes >> >> > allocated >> >> > ==2371== >> >> > ==2371== LEAK SUMMARY: >> >> > ==2371== definitely lost: 26 bytes in 1 blocks >> >> > ==2371== indirectly lost: 0 bytes in 0 blocks >> >> > ==2371== possibly lost: 88 bytes in 2 blocks >> >> > ==2371== still reachable: 209,008 bytes in 2,118 blocks >> >> > ==2371== suppressed: 0 bytes in 0 blocks >> >> > ==2371== Rerun with --leak-check=full to see details of leaked memory >> >> > ==2371== >> >> > ==2371== For counts of detected and suppressed errors, rerun with: -v >> >> > ==2371== Use --track-origins=yes to see where uninitialised values >> come >> >> from >> >> > ==2371== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 114 >> from 7) >> >> > [root valgrind-3.11.0]# ==2375== Thread 6 rs:main Q:Reg: >> >> > ==2375== Invalid read of size 8 >> >> > ==2375== at 0x56607BE: fjson_object_iter_next >> >> > (json_object_iterator.c:119) >> >> > ==2375== by 0x5660863: fjson_object_iter_begin >> >> > (json_object_iterator.c:77) >> >> > ==2375== by 0x126F09: ??? (in /sbin/rsyslogd) >> >> > ==2375== by 0x1270A0: msgAddJSON (in /sbin/rsyslogd) >> >> > ==2375== by 0x7242F2D: ??? (in /lib64/rsyslog/mmnormalize.so) >> >> > ==2375== by 0x14EA64: ??? (in /sbin/rsyslogd) >> >> > ==2375== by 0x14F561: ??? (in /sbin/rsyslogd) >> >> > ==2375== by 0x14F923: ??? (in /sbin/rsyslogd) >> >> > ==2375== by 0x146AB6: ??? (in /sbin/rsyslogd) >> >> > ==2375== by 0x1473DA: ??? (in /sbin/rsyslogd) >> >> > ==2375== by 0x156FE3: ??? (in /sbin/rsyslogd) >> >> > ==2375== by 0x1457D8: ??? (in /sbin/rsyslogd) >> >> > ==2375== Address 0xf57d338 is 0 bytes after a block of size 72 >> alloc'd >> >> > ==2375== at 0x4A05FEF: calloc (vg_replace_malloc.c:711) >> >> > ==2375== by 0x7659871: json_object_new (json_object.c:178) >> >> > ==2375== by 0x765A61A: json_object_new_object (json_object.c:354) >> >> > ==2375== by 0x744A355: ln_normalize (in >> >> /usr/lib64/liblognorm.so.2.0.0) >> >> > ==2375== by 0x7242EEB: ??? (in /lib64/rsyslog/mmnormalize.so) >> >> > ==2375== by 0x14EA64: ??? (in /sbin/rsyslogd) >> >> > ==2375== by 0x14F561: ??? (in /sbin/rsyslogd) >> >> > ==2375== by 0x14F923: ??? (in /sbin/rsyslogd) >> >> > ==2375== by 0x146AB6: ??? (in /sbin/rsyslogd) >> >> > ==2375== by 0x1473DA: ??? (in /sbin/rsyslogd) >> >> > ==2375== by 0x156FE3: ??? (in /sbin/rsyslogd) >> >> > ==2375== by 0x1457D8: ??? (in /sbin/rsyslogd) >> >> > ==2375== >> >> > ==2375== Invalid read of size 4 >> >> > ==2375== at 0x5660778: fjson_object_get (json_object.c:176) >> >> > ==2375== by 0x126F57: ??? (in /sbin/rsyslogd) >> >> > ==2375== by 0x1270A0: msgAddJSON (in /sbin/rsyslogd) >> >> > ==2375== by 0x7242F2D: ??? (in /lib64/rsyslog/mmnormalize.so) >> >> > ==2375== by 0x14EA64: ??? (in /sbin/rsyslogd) >> >> > ==2375== by 0x14F561: ??? (in /sbin/rsyslogd) >> >> > ==2375== by 0x14F923: ??? (in /sbin/rsyslogd) >> >> > ==2375== by 0x146AB6: ??? (in /sbin/rsyslogd) >> >> > ==2375== by 0x1473DA: ??? (in /sbin/rsyslogd) >> >> > ==2375== by 0x156FE3: ??? (in /sbin/rsyslogd) >> >> > ==2375== by 0x1457D8: ??? (in /sbin/rsyslogd) >> >> > ==2375== by 0x13FB5D: wtiWorker (in /sbin/rsyslogd) >> >> > ==2375== Address 0x60000001e is not stack'd, malloc'd or (recently) >> >> free'd >> >> > ==2375== >> >> > ==2375== >> >> > ==2375== Process terminating with default action of signal 11 >> (SIGSEGV) >> >> > ==2375== Access not within mapped region at address 0x60000001E >> >> > ==2375== at 0x5660778: fjson_object_get (json_object.c:176) >> >> > ==2375== by 0x126F57: ??? (in /sbin/rsyslogd) >> >> > ==2375== by 0x1270A0: msgAddJSON (in /sbin/rsyslogd) >> >> > ==2375== by 0x7242F2D: ??? (in /lib64/rsyslog/mmnormalize.so) >> >> > ==2375== by 0x14EA64: ??? (in /sbin/rsyslogd) >> >> > ==2375== by 0x14F561: ??? (in /sbin/rsyslogd) >> >> > ==2375== by 0x14F923: ??? (in /sbin/rsyslogd) >> >> > ==2375== by 0x146AB6: ??? (in /sbin/rsyslogd) >> >> > ==2375== by 0x1473DA: ??? (in /sbin/rsyslogd) >> >> > ==2375== by 0x156FE3: ??? (in /sbin/rsyslogd) >> >> > ==2375== by 0x1457D8: ??? (in /sbin/rsyslogd) >> >> > ==2375== by 0x13FB5D: wtiWorker (in /sbin/rsyslogd) >> >> > ==2375== If you believe this happened as a result of a stack >> >> > ==2375== overflow in your program's main thread (unlikely but >> >> > ==2375== possible), you can try to increase the size of the >> >> > ==2375== main thread stack using the --main-stacksize= flag. >> >> > ==2375== The main thread stack size used in this run was 10485760. >> >> > ==2375== >> >> > ==2375== HEAP SUMMARY: >> >> > ==2375== in use at exit: 2,611,014 bytes in 11,732 blocks >> >> > ==2375== total heap usage: 27,013 allocs, 15,281 frees, 8,524,299 >> bytes >> >> > allocated >> >> > ==2375== >> >> > ==2375== LEAK SUMMARY: >> >> > ==2375== definitely lost: 4,117 bytes in 11 blocks >> >> > ==2375== indirectly lost: 24,922 bytes in 20 blocks >> >> > ==2375== possibly lost: 4,464 bytes in 12 blocks >> >> > ==2375== still reachable: 2,577,511 bytes in 11,689 blocks >> >> > ==2375== suppressed: 0 bytes in 0 blocks >> >> > ==2375== Rerun with --leak-check=full to see details of leaked memory >> >> > ==2375== >> >> > ==2375== For counts of detected and suppressed errors, rerun with: -v >> >> > ==2375== Use --track-origins=yes to see where uninitialised values >> come >> >> from >> >> > ==2375== ERROR SUMMARY: 5 errors from 3 contexts (suppressed: 120 >> from 7) >> >> > >> >> > [root valgrind-3.11.0]# rsyslogd -v >> >> > rsyslogd 8.20.0, compiled with: >> >> > PLATFORM: x86_64-redhat-linux-gnu >> >> > PLATFORM (lsb_release -d): >> >> > FEATURE_REGEXP: Yes >> >> > GSSAPI Kerberos 5 support: No >> >> > FEATURE_DEBUG (debug build, slow code): No >> >> > 32bit Atomic operations supported: Yes >> >> > 64bit Atomic operations supported: Yes >> >> > memory allocator: system default >> >> > Runtime Instrumentation (slow code): No >> >> > uuid support: Yes >> >> > Number of Bits in RainerScript integers: 64 >> >> > >> >> > See http://www.rsyslog.com for more information. >> >> > >> >> > [root valgrind-3.11.0]# dmesg >> >> > .. >> >> > rs:main Q:Reg[1776]: segfault at 65 ip 00007f0769bcc2b6 sp >> >> 00007f0762de87e8 >> >> > error 4 in libc-2.12.so[7f0769aa4000+18a000] >> >> > >> >> > Thanks, >> >> > >> >> > Alec >> >> > >> >> > On Wed, Jul 20, 2016 at 11:32 PM, Rainer Gerhards < >> >> [email protected]> >> >> > wrote: >> >> > >> >> >> 2016-07-20 23:02 GMT+02:00 Alec Swan <[email protected]>: >> >> >> > I used test RPMs to upgrade from 8.19.0 to 8.20.0. However, now >> when I >> >> >> > start rsyslog it logs the following error message to dmesg and >> shuts >> >> >> > rsyslog down: >> >> >> > "rs:main Q:Reg[23801]: segfault at 1a ip 00007fa95ac9f778 sp >> >> >> > 00007fa94f5fd878 error 6 in >> libfastjson.so.4.0.0[7fa95ac9c000+9000]" >> >> >> > >> >> >> > These are the packages that were installed/updated: >> >> >> > Jul 20 20:53:11 yum[22693]: Installed: >> >> libfastjson4-0.99.3-2.el6.x86_64 >> >> >> > Jul 20 20:53:11 yum[22693]: Updated: rsyslog-8.20.0-1.el6.x86_64 >> >> >> > Jul 20 20:53:11 yum[22693]: Updated: >> >> >> rsyslog-mmutf8fix-8.20.0-1.el6.x86_64 >> >> >> > Jul 20 20:53:11 yum[22693]: Updated: >> >> >> rsyslog-mmnormalize-8.20.0-1.el6.x86_64 >> >> >> > Jul 20 20:53:11 yum[22693]: Updated: >> >> >> > rsyslog-elasticsearch-8.20.0-1.el6.x86_64 >> >> >> >> >> >> would it be possible that you do a run under valgrind control? >> >> >> >> >> >> Rainer >> >> >> > >> >> >> > >> >> >> > Thanks, >> >> >> > >> >> >> > Alec >> >> >> > >> >> >> > On Wed, Jul 20, 2016 at 12:11 PM, Michael Biebl <[email protected]> >> >> >> wrote: >> >> >> > >> >> >> >> 2016-07-12 23:12 GMT+02:00 David Lang <[email protected]>: >> >> >> >> > On Tue, 12 Jul 2016, Michael Biebl wrote: >> >> >> >> > >> >> >> >> >> Fwiw, this is the first release I felt comfortable enough >> >> uploading >> >> >> to >> >> >> >> >> Debian unstable. >> >> >> >> >> It just hit the archive a few hours ago: >> >> >> >> >> https://tracker.debian.org/pkg/libfastjson >> >> >> >> >> >> >> >> >> >> I hope we don't break the API that much anymore before 1.0.0. >> >> >> >> >> >> >> >> >> >> I'm now waiting for a liblognorm 1.1.4 release which compiles >> >> against >> >> >> >> >> libfastjson 0.99.3. >> >> >> >> > >> >> >> >> > >> >> >> >> > My understanding is that we are going to get liblognorm 2 this >> >> cycle >> >> >> >> (after >> >> >> >> > being delayed for the last couple of cycles) >> >> >> >> > >> >> >> >> >> >> >> >> >> >> >> >> Ok, so what's the plan here? Which liblognorm version should I >> pick? >> >> >> >> At least they currently released version doesn't compile against >> the >> >> >> >> latest libfastjson release (which is not a surprise, after the API >> >> >> >> rework, which I'm thankful for). >> >> >> >> >> >> >> >> Regards, >> >> >> >> Michael >> >> >> >> >> >> >> >> -- >> >> >> >> Why is it that all of the instruments seeking intelligent life in >> the >> >> >> >> universe are pointed away from Earth? >> >> >> >> _______________________________________________ >> >> >> >> rsyslog mailing list >> >> >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> >> >> http://www.rsyslog.com/professional-services/ >> >> >> >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >> >> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> >> myriad >> >> >> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >> >> you >> >> >> >> DON'T LIKE THAT. >> >> >> >> >> >> >> > _______________________________________________ >> >> >> > rsyslog mailing list >> >> >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> >> > http://www.rsyslog.com/professional-services/ >> >> >> > What's up with rsyslog? Follow https://twitter.com/rgerhards >> >> >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> >> myriad >> >> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >> you >> >> >> DON'T LIKE THAT. >> >> >> _______________________________________________ >> >> >> rsyslog mailing list >> >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> >> http://www.rsyslog.com/professional-services/ >> >> >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> myriad >> >> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >> you >> >> >> DON'T LIKE THAT. >> >> >> >> >> > _______________________________________________ >> >> > rsyslog mailing list >> >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> > http://www.rsyslog.com/professional-services/ >> >> > What's up with rsyslog? Follow https://twitter.com/rgerhards >> >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> myriad >> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> >> DON'T LIKE THAT. >> >> _______________________________________________ >> >> rsyslog mailing list >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> http://www.rsyslog.com/professional-services/ >> >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> >> DON'T LIKE THAT. >> >> >> > >> > _______________________________________________ >> > rsyslog mailing list >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > http://www.rsyslog.com/professional-services/ >> > What's up with rsyslog? Follow https://twitter.com/rgerhards >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
