2016-07-26 17:35 GMT+02:00 Alec Swan <[email protected]>: > Rainer, per Florian's request in "experimental rpms for rsyslog-8.20.0" I > downloaded RPMs from the "testing" repo baseurl= > http://rpms.adiscon.com/testing/epel-$releasever/$basearch
Can you check where the system says it actually installed them from. The line numbers do not match the RPMs Florian created, thus I ask. Rainer > > > Thanks, > > Alec > > On Tue, Jul 26, 2016 at 7:30 AM, Rainer Gerhards <[email protected]> > wrote: > >> I today tried quite a while to reproduce the issue with your config, >> unfortunately without any success. For now, I conclude that this is >> not a general problem, but one induced by your environment and data. >> >> I have then looked again at the traces and the few line numbers it >> contains. Now comes the interesting fact. I notice that there is a >> mismatch between the line numbers shown and the code that actually was >> used to build the packages (judging from git and the source RPMs). So >> I wonder what might have gone wrong here. Can you please let me know >> which exact repro you installed from and potentially which other repos >> you have configured on that machine. It might also be useful to query >> the system from where exactly the packages have ben installed. I found >> [1], which hopefully helps to get that information. >> >> Once I have this information, I can check if there are any version >> quirks (note that commit [2] fixed a situation that could cause >> exactly what you see). If there is no issue with the package sources, >> I unfortunately need your help in further troubleshooting this in your >> environment. But let's first see what the repo check will bring up. >> >> Rainer >> >> [1] >> http://serverfault.com/questions/62026/how-to-know-from-which-yum-repository-a-package-has-been-installed >> [2] >> https://github.com/rsyslog/libfastjson/commit/0499ec8df09c913156ead426c171031b566dbe7d >> >> 2016-07-23 19:28 GMT+02:00 Alec Swan <[email protected]>: >> > Yes, that should be OK. Is there a link to the build server/CI env I can >> > use? >> > >> > Thanks, >> > >> > Alec >> > >> > On Sat, Jul 23, 2016 at 7:08 AM, Rainer Gerhards < >> [email protected]> >> > wrote: >> > >> >> Thanks. Would it be OK for your if a somewhat mangled version of these >> >> configs show up inside the testbench? I think about adding a specific >> >> test for it, that would be great for the future as well. I'd probably >> >> try to strip it a bit down (e.g. no ES but writing to file, which >> >> would enable runs on all CI environments and when no ES is available >> >> -- I assume this would also trigger the bug). >> >> >> >> Please let me know, >> >> Rainer >> >> >> >> 2016-07-22 17:46 GMT+02:00 Alec Swan <[email protected]>: >> >> > Rainer, I attached /etc/rsyslog.conf and >> >> /etc/rsyslog.d/rsyslog-custom.conf >> >> > >> >> > Thanks, >> >> > >> >> > Alec >> >> > >> >> > On Fri, Jul 22, 2016 at 12:42 AM, Rainer Gerhards < >> >> [email protected]> >> >> > wrote: >> >> > >> >> >> unfortunately, we do not seem to have debug symbols, so I don't see >> >> >> much more. Can you share your full config (including all includes, if >> >> >> any)? Maybe I can get it to abort in my lab as well. >> >> >> >> >> >> Rainer >> >> >> >> >> >> 2016-07-21 22:02 GMT+02:00 Alec Swan <[email protected]>: >> >> >> > Rainer, here is the output from running rsyslog under valgrind: >> >> >> > >> >> >> > [root valgrind-3.11.0]# valgrind rsyslogd >> >> >> > ==2371== Memcheck, a memory error detector >> >> >> > ==2371== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward >> et >> >> al. >> >> >> > ==2371== Using Valgrind-3.11.0 and LibVEX; rerun with -h for >> copyright >> >> >> info >> >> >> > ==2371== Command: rsyslogd >> >> >> > ==2371== >> >> >> > ==2371== Conditional jump or move depends on uninitialised value(s) >> >> >> > ==2371== at 0x16A414: cnfexprDestruct (in /sbin/rsyslogd) >> >> >> > ==2371== by 0x16B36D: cnfstmtOptimize (in /sbin/rsyslogd) >> >> >> > ==2371== by 0x146317: ??? (in /sbin/rsyslogd) >> >> >> > ==2371== by 0x12E5BE: llExecFunc (in /sbin/rsyslogd) >> >> >> > ==2371== by 0x146176: rulesetOptimizeAll (in /sbin/rsyslogd) >> >> >> > ==2371== by 0x123EF6: ??? (in /sbin/rsyslogd) >> >> >> > ==2371== by 0x158A68: ??? (in /sbin/rsyslogd) >> >> >> > ==2371== by 0x159008: main (in /sbin/rsyslogd) >> >> >> > ==2371== >> >> >> > ==2375== Warning: invalid file descriptor 99990 in syscall close() >> >> >> > ==2371== >> >> >> > ==2371== HEAP SUMMARY: >> >> >> > ==2371== in use at exit: 209,122 bytes in 2,121 blocks >> >> >> > ==2371== total heap usage: 11,004 allocs, 8,883 frees, 4,319,525 >> >> bytes >> >> >> > allocated >> >> >> > ==2371== >> >> >> > ==2371== LEAK SUMMARY: >> >> >> > ==2371== definitely lost: 26 bytes in 1 blocks >> >> >> > ==2371== indirectly lost: 0 bytes in 0 blocks >> >> >> > ==2371== possibly lost: 88 bytes in 2 blocks >> >> >> > ==2371== still reachable: 209,008 bytes in 2,118 blocks >> >> >> > ==2371== suppressed: 0 bytes in 0 blocks >> >> >> > ==2371== Rerun with --leak-check=full to see details of leaked >> memory >> >> >> > ==2371== >> >> >> > ==2371== For counts of detected and suppressed errors, rerun with: >> -v >> >> >> > ==2371== Use --track-origins=yes to see where uninitialised values >> >> come >> >> >> from >> >> >> > ==2371== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 114 >> >> from 7) >> >> >> > [root valgrind-3.11.0]# ==2375== Thread 6 rs:main Q:Reg: >> >> >> > ==2375== Invalid read of size 8 >> >> >> > ==2375== at 0x56607BE: fjson_object_iter_next >> >> >> > (json_object_iterator.c:119) >> >> >> > ==2375== by 0x5660863: fjson_object_iter_begin >> >> >> > (json_object_iterator.c:77) >> >> >> > ==2375== by 0x126F09: ??? (in /sbin/rsyslogd) >> >> >> > ==2375== by 0x1270A0: msgAddJSON (in /sbin/rsyslogd) >> >> >> > ==2375== by 0x7242F2D: ??? (in /lib64/rsyslog/mmnormalize.so) >> >> >> > ==2375== by 0x14EA64: ??? (in /sbin/rsyslogd) >> >> >> > ==2375== by 0x14F561: ??? (in /sbin/rsyslogd) >> >> >> > ==2375== by 0x14F923: ??? (in /sbin/rsyslogd) >> >> >> > ==2375== by 0x146AB6: ??? (in /sbin/rsyslogd) >> >> >> > ==2375== by 0x1473DA: ??? (in /sbin/rsyslogd) >> >> >> > ==2375== by 0x156FE3: ??? (in /sbin/rsyslogd) >> >> >> > ==2375== by 0x1457D8: ??? (in /sbin/rsyslogd) >> >> >> > ==2375== Address 0xf57d338 is 0 bytes after a block of size 72 >> >> alloc'd >> >> >> > ==2375== at 0x4A05FEF: calloc (vg_replace_malloc.c:711) >> >> >> > ==2375== by 0x7659871: json_object_new (json_object.c:178) >> >> >> > ==2375== by 0x765A61A: json_object_new_object >> (json_object.c:354) >> >> >> > ==2375== by 0x744A355: ln_normalize (in >> >> >> /usr/lib64/liblognorm.so.2.0.0) >> >> >> > ==2375== by 0x7242EEB: ??? (in /lib64/rsyslog/mmnormalize.so) >> >> >> > ==2375== by 0x14EA64: ??? (in /sbin/rsyslogd) >> >> >> > ==2375== by 0x14F561: ??? (in /sbin/rsyslogd) >> >> >> > ==2375== by 0x14F923: ??? (in /sbin/rsyslogd) >> >> >> > ==2375== by 0x146AB6: ??? (in /sbin/rsyslogd) >> >> >> > ==2375== by 0x1473DA: ??? (in /sbin/rsyslogd) >> >> >> > ==2375== by 0x156FE3: ??? (in /sbin/rsyslogd) >> >> >> > ==2375== by 0x1457D8: ??? (in /sbin/rsyslogd) >> >> >> > ==2375== >> >> >> > ==2375== Invalid read of size 4 >> >> >> > ==2375== at 0x5660778: fjson_object_get (json_object.c:176) >> >> >> > ==2375== by 0x126F57: ??? (in /sbin/rsyslogd) >> >> >> > ==2375== by 0x1270A0: msgAddJSON (in /sbin/rsyslogd) >> >> >> > ==2375== by 0x7242F2D: ??? (in /lib64/rsyslog/mmnormalize.so) >> >> >> > ==2375== by 0x14EA64: ??? (in /sbin/rsyslogd) >> >> >> > ==2375== by 0x14F561: ??? (in /sbin/rsyslogd) >> >> >> > ==2375== by 0x14F923: ??? (in /sbin/rsyslogd) >> >> >> > ==2375== by 0x146AB6: ??? (in /sbin/rsyslogd) >> >> >> > ==2375== by 0x1473DA: ??? (in /sbin/rsyslogd) >> >> >> > ==2375== by 0x156FE3: ??? (in /sbin/rsyslogd) >> >> >> > ==2375== by 0x1457D8: ??? (in /sbin/rsyslogd) >> >> >> > ==2375== by 0x13FB5D: wtiWorker (in /sbin/rsyslogd) >> >> >> > ==2375== Address 0x60000001e is not stack'd, malloc'd or >> (recently) >> >> >> free'd >> >> >> > ==2375== >> >> >> > ==2375== >> >> >> > ==2375== Process terminating with default action of signal 11 >> >> (SIGSEGV) >> >> >> > ==2375== Access not within mapped region at address 0x60000001E >> >> >> > ==2375== at 0x5660778: fjson_object_get (json_object.c:176) >> >> >> > ==2375== by 0x126F57: ??? (in /sbin/rsyslogd) >> >> >> > ==2375== by 0x1270A0: msgAddJSON (in /sbin/rsyslogd) >> >> >> > ==2375== by 0x7242F2D: ??? (in /lib64/rsyslog/mmnormalize.so) >> >> >> > ==2375== by 0x14EA64: ??? (in /sbin/rsyslogd) >> >> >> > ==2375== by 0x14F561: ??? (in /sbin/rsyslogd) >> >> >> > ==2375== by 0x14F923: ??? (in /sbin/rsyslogd) >> >> >> > ==2375== by 0x146AB6: ??? (in /sbin/rsyslogd) >> >> >> > ==2375== by 0x1473DA: ??? (in /sbin/rsyslogd) >> >> >> > ==2375== by 0x156FE3: ??? (in /sbin/rsyslogd) >> >> >> > ==2375== by 0x1457D8: ??? (in /sbin/rsyslogd) >> >> >> > ==2375== by 0x13FB5D: wtiWorker (in /sbin/rsyslogd) >> >> >> > ==2375== If you believe this happened as a result of a stack >> >> >> > ==2375== overflow in your program's main thread (unlikely but >> >> >> > ==2375== possible), you can try to increase the size of the >> >> >> > ==2375== main thread stack using the --main-stacksize= flag. >> >> >> > ==2375== The main thread stack size used in this run was 10485760. >> >> >> > ==2375== >> >> >> > ==2375== HEAP SUMMARY: >> >> >> > ==2375== in use at exit: 2,611,014 bytes in 11,732 blocks >> >> >> > ==2375== total heap usage: 27,013 allocs, 15,281 frees, 8,524,299 >> >> bytes >> >> >> > allocated >> >> >> > ==2375== >> >> >> > ==2375== LEAK SUMMARY: >> >> >> > ==2375== definitely lost: 4,117 bytes in 11 blocks >> >> >> > ==2375== indirectly lost: 24,922 bytes in 20 blocks >> >> >> > ==2375== possibly lost: 4,464 bytes in 12 blocks >> >> >> > ==2375== still reachable: 2,577,511 bytes in 11,689 blocks >> >> >> > ==2375== suppressed: 0 bytes in 0 blocks >> >> >> > ==2375== Rerun with --leak-check=full to see details of leaked >> memory >> >> >> > ==2375== >> >> >> > ==2375== For counts of detected and suppressed errors, rerun with: >> -v >> >> >> > ==2375== Use --track-origins=yes to see where uninitialised values >> >> come >> >> >> from >> >> >> > ==2375== ERROR SUMMARY: 5 errors from 3 contexts (suppressed: 120 >> >> from 7) >> >> >> > >> >> >> > [root valgrind-3.11.0]# rsyslogd -v >> >> >> > rsyslogd 8.20.0, compiled with: >> >> >> > PLATFORM: x86_64-redhat-linux-gnu >> >> >> > PLATFORM (lsb_release -d): >> >> >> > FEATURE_REGEXP: Yes >> >> >> > GSSAPI Kerberos 5 support: No >> >> >> > FEATURE_DEBUG (debug build, slow code): No >> >> >> > 32bit Atomic operations supported: Yes >> >> >> > 64bit Atomic operations supported: Yes >> >> >> > memory allocator: system default >> >> >> > Runtime Instrumentation (slow code): No >> >> >> > uuid support: Yes >> >> >> > Number of Bits in RainerScript integers: 64 >> >> >> > >> >> >> > See http://www.rsyslog.com for more information. >> >> >> > >> >> >> > [root valgrind-3.11.0]# dmesg >> >> >> > .. >> >> >> > rs:main Q:Reg[1776]: segfault at 65 ip 00007f0769bcc2b6 sp >> >> >> 00007f0762de87e8 >> >> >> > error 4 in libc-2.12.so[7f0769aa4000+18a000] >> >> >> > >> >> >> > Thanks, >> >> >> > >> >> >> > Alec >> >> >> > >> >> >> > On Wed, Jul 20, 2016 at 11:32 PM, Rainer Gerhards < >> >> >> [email protected]> >> >> >> > wrote: >> >> >> > >> >> >> >> 2016-07-20 23:02 GMT+02:00 Alec Swan <[email protected]>: >> >> >> >> > I used test RPMs to upgrade from 8.19.0 to 8.20.0. However, now >> >> when I >> >> >> >> > start rsyslog it logs the following error message to dmesg and >> >> shuts >> >> >> >> > rsyslog down: >> >> >> >> > "rs:main Q:Reg[23801]: segfault at 1a ip 00007fa95ac9f778 sp >> >> >> >> > 00007fa94f5fd878 error 6 in >> >> libfastjson.so.4.0.0[7fa95ac9c000+9000]" >> >> >> >> > >> >> >> >> > These are the packages that were installed/updated: >> >> >> >> > Jul 20 20:53:11 yum[22693]: Installed: >> >> >> libfastjson4-0.99.3-2.el6.x86_64 >> >> >> >> > Jul 20 20:53:11 yum[22693]: Updated: rsyslog-8.20.0-1.el6.x86_64 >> >> >> >> > Jul 20 20:53:11 yum[22693]: Updated: >> >> >> >> rsyslog-mmutf8fix-8.20.0-1.el6.x86_64 >> >> >> >> > Jul 20 20:53:11 yum[22693]: Updated: >> >> >> >> rsyslog-mmnormalize-8.20.0-1.el6.x86_64 >> >> >> >> > Jul 20 20:53:11 yum[22693]: Updated: >> >> >> >> > rsyslog-elasticsearch-8.20.0-1.el6.x86_64 >> >> >> >> >> >> >> >> would it be possible that you do a run under valgrind control? >> >> >> >> >> >> >> >> Rainer >> >> >> >> > >> >> >> >> > >> >> >> >> > Thanks, >> >> >> >> > >> >> >> >> > Alec >> >> >> >> > >> >> >> >> > On Wed, Jul 20, 2016 at 12:11 PM, Michael Biebl < >> [email protected]> >> >> >> >> wrote: >> >> >> >> > >> >> >> >> >> 2016-07-12 23:12 GMT+02:00 David Lang <[email protected]>: >> >> >> >> >> > On Tue, 12 Jul 2016, Michael Biebl wrote: >> >> >> >> >> > >> >> >> >> >> >> Fwiw, this is the first release I felt comfortable enough >> >> >> uploading >> >> >> >> to >> >> >> >> >> >> Debian unstable. >> >> >> >> >> >> It just hit the archive a few hours ago: >> >> >> >> >> >> https://tracker.debian.org/pkg/libfastjson >> >> >> >> >> >> >> >> >> >> >> >> I hope we don't break the API that much anymore before >> 1.0.0. >> >> >> >> >> >> >> >> >> >> >> >> I'm now waiting for a liblognorm 1.1.4 release which >> compiles >> >> >> against >> >> >> >> >> >> libfastjson 0.99.3. >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> > My understanding is that we are going to get liblognorm 2 >> this >> >> >> cycle >> >> >> >> >> (after >> >> >> >> >> > being delayed for the last couple of cycles) >> >> >> >> >> > >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> Ok, so what's the plan here? Which liblognorm version should I >> >> pick? >> >> >> >> >> At least they currently released version doesn't compile >> against >> >> the >> >> >> >> >> latest libfastjson release (which is not a surprise, after the >> API >> >> >> >> >> rework, which I'm thankful for). >> >> >> >> >> >> >> >> >> >> Regards, >> >> >> >> >> Michael >> >> >> >> >> >> >> >> >> >> -- >> >> >> >> >> Why is it that all of the instruments seeking intelligent life >> in >> >> the >> >> >> >> >> universe are pointed away from Earth? >> >> >> >> >> _______________________________________________ >> >> >> >> >> rsyslog mailing list >> >> >> >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> >> >> >> http://www.rsyslog.com/professional-services/ >> >> >> >> >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >> >> >> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED >> by a >> >> >> myriad >> >> >> >> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT >> POST if >> >> >> you >> >> >> >> >> DON'T LIKE THAT. >> >> >> >> >> >> >> >> >> > _______________________________________________ >> >> >> >> > rsyslog mailing list >> >> >> >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> >> >> > http://www.rsyslog.com/professional-services/ >> >> >> >> > What's up with rsyslog? Follow https://twitter.com/rgerhards >> >> >> >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by >> a >> >> >> myriad >> >> >> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >> >> you >> >> >> >> DON'T LIKE THAT. >> >> >> >> _______________________________________________ >> >> >> >> rsyslog mailing list >> >> >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> >> >> http://www.rsyslog.com/professional-services/ >> >> >> >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >> >> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> >> myriad >> >> >> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >> >> you >> >> >> >> DON'T LIKE THAT. >> >> >> >> >> >> >> > _______________________________________________ >> >> >> > rsyslog mailing list >> >> >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> >> > http://www.rsyslog.com/professional-services/ >> >> >> > What's up with rsyslog? Follow https://twitter.com/rgerhards >> >> >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> >> myriad >> >> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >> you >> >> >> DON'T LIKE THAT. >> >> >> _______________________________________________ >> >> >> rsyslog mailing list >> >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> >> http://www.rsyslog.com/professional-services/ >> >> >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> myriad >> >> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >> you >> >> >> DON'T LIKE THAT. >> >> >> >> >> > >> >> > _______________________________________________ >> >> > rsyslog mailing list >> >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> > http://www.rsyslog.com/professional-services/ >> >> > What's up with rsyslog? Follow https://twitter.com/rgerhards >> >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> myriad >> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> >> DON'T LIKE THAT. >> >> _______________________________________________ >> >> rsyslog mailing list >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> >> http://www.rsyslog.com/professional-services/ >> >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> >> DON'T LIKE THAT. >> >> >> > _______________________________________________ >> > rsyslog mailing list >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > http://www.rsyslog.com/professional-services/ >> > What's up with rsyslog? Follow https://twitter.com/rgerhards >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
