Thanks, David. I think the problem is with the old omelasticsearch-queue.qi which was created back when rsyslog had problems with on-disk queues.
What are the next steps for releasing 8.20 RPMs? When are you guys planning to upload them to production yum repo? Thanks, Alec On Wed, Jul 27, 2016 at 12:51 PM, David Lang <[email protected]> wrote: > I'm running 8.21 (self compiled) with all the latest libraries and am not > having problems with queues. > > the error you are getting (fatal error on disk queue 'action 5 queue[DA]', > emergency switch to direct mode) is one I've run across when there are > permission problems with the directory that the queue is in, or if there > are old queue files and no .qi file > > David Lang > > On Wed, 27 Jul 2016, Alec Swan wrote: > > Date: Wed, 27 Jul 2016 12:10:32 -0600 >> From: Alec Swan <[email protected]> >> Reply-To: rsyslog-users <[email protected]> >> To: rsyslog-users <[email protected]> >> Subject: Re: [rsyslog] libfastjson 0.99.3 released >> >> Rainer, per your request I re-ran the test and was able to start rsyslog >> with the newly downloaded binaries. The following shows test output. I >> also >> included the logs from /var/log/messages which indicate that there might >> be >> a problem with some queues. Could you please confirm that this is not a >> regression of the queue related issues fixed in 8.18.0 or 8.19.0? >> >> # yum erase rsyslog-mmutf8fix rsyslog-elasticsearch rsyslog-mmnormalize >> liblognorm1 rsyslog json-c libfastjson4 >> ... >> Removed: >> json-c.x86_64 0:0.11-11.el6 libfastjson4.x86_64 >> 0:0.99.3-2.el6 liblognorm1.x86_64 0:1.1.3-1.el6 >> rsyslog.x86_64 0:8.20.0-1.el6 >> rsyslog-elasticsearch.x86_64 0:8.20.0-1.el6 >> rsyslog-mmnormalize.x86_64 0:8.20.0-1.el6 rsyslog-mmutf8fix.x86_64 >> 0:8.20.0-1.el6 >> >> Complete! >> >> # rpm -qa | grep rsys >> >> # yumdownloader --resolve rsyslog-mmnormalize-8.20.0 >> rsyslog-elasticsearch-8.20.0 rsyslog-mmutf8fix-8.20.0 >> ... >> rsyslog_v8 >> >> | 2.5 kB 00:00 >> rsyslog_v8/primary_db >> >> | 264 kB 00:00 >> --> Running transaction check >> ---> Package rsyslog-elasticsearch.x86_64 0:8.20.0-1.el6 will be installed >> --> Processing Dependency: rsyslog = 8.20.0-1.el6 for package: >> rsyslog-elasticsearch-8.20.0-1.el6.x86_64 >> ---> Package rsyslog-mmnormalize.x86_64 0:8.20.0-1.el6 will be installed >> --> Processing Dependency: liblognorm.so.5()(64bit) for package: >> rsyslog-mmnormalize-8.20.0-1.el6.x86_64 >> ---> Package rsyslog-mmutf8fix.x86_64 0:8.20.0-1.el6 will be installed >> --> Running transaction check >> ---> Package liblognorm2.x86_64 0:2.0.0-1.el6 will be installed >> --> Processing Dependency: libfastjson.so.4()(64bit) for package: >> liblognorm2-2.0.0-1.el6.x86_64 >> ---> Package rsyslog.x86_64 0:8.20.0-1.el6 will be installed >> --> Running transaction check >> ---> Package libfastjson4.x86_64 0:0.99.3-2.el6 will be installed >> --> Finished Dependency Resolution >> rsyslog-mmnormalize-8.20.0-1.el6.x86_64.rpm >> >> | 13 kB 00:00 >> rsyslog-elasticsearch-8.20.0-1.el6.x86_64.rpm >> >> | 24 kB 00:00 >> rsyslog-mmutf8fix-8.20.0-1.el6.x86_64.rpm >> >> | 12 kB 00:00 >> rsyslog-8.20.0-1.el6.x86_64.rpm >> >> | 691 kB 00:00 >> liblognorm2-2.0.0-1.el6.x86_64.rpm >> >> | 70 kB 00:00 >> libfastjson4-0.99.3-2.el6.x86_64.rpm >> >> | 46 kB 00:00 >> >> # ls >> libfastjson4-0.99.3-2.el6.x86_64.rpm rsyslog-8.20.0-1.el6.x86_64.rpm >> rsyslog-mmnormalize-8.20.0-1.el6.x86_64.rpm >> liblognorm2-2.0.0-1.el6.x86_64.rpm >> rsyslog-elasticsearch-8.20.0-1.el6.x86_64.rpm >> rsyslog-mmutf8fix-8.20.0-1.el6.x86_64.rpm >> >> # rpm -ivh *.rpm >> warning: libfastjson4-0.99.3-2.el6.x86_64.rpm: Header V3 RSA/SHA1 >> Signature, key ID e00b8985: NOKEY >> Preparing... ########################################### >> [100%] >> 1:libfastjson4 ########################################### [ >> 17%] >> 2:rsyslog ########################################### [ >> 33%] >> 3:liblognorm2 ########################################### [ >> 50%] >> 4:rsyslog-mmnormalize ########################################### [ >> 67%] >> 5:rsyslog-elasticsearch ########################################### [ >> 83%] >> 6:rsyslog-mmutf8fix ########################################### >> [100%] >> >> # ps -ax | grep rsyslog >> Warning: bad syntax, perhaps a bogus '-'? See >> /usr/share/doc/procps-3.2.8/FAQ >> 3494 pts/0 S+ 0:00 grep rsyslog >> >> # service rsyslog start >> Starting system logger: [ OK ] >> >> # ps -ax | grep rsyslog >> Warning: bad syntax, perhaps a bogus '-'? See >> /usr/share/doc/procps-3.2.8/FAQ >> 3578 ? Ssl 0:00 /sbin/rsyslogd -i /var/run/syslogd.pid >> 3602 pts/0 S+ 0:00 grep rsyslog >> >> # tail /var/log/messages >> Jul 27 18:06:10 rsyslogd: [origin software="rsyslogd" swVersion="8.20.0" >> x-pid="3526" x-info="http://www.rsyslog.com";] exiting on signal 15. >> Jul 27 18:06:11 rsyslogd: [origin software="rsyslogd" swVersion="8.20.0" >> x-pid="3578" x-info="http://www.rsyslog.com";] start >> Jul 27 18:06:11 rsyslogd-2040: fatal error on disk queue 'action 5 >> queue[DA]', emergency switch to direct mode [v8.20.0 try >> http://www.rsyslog.com/e/2040 ] >> Jul 27 18:06:11 rsyslogd-2040: fatal error on disk queue 'action 3 >> queue[DA]', emergency switch to direct mode [v8.20.0 try >> http://www.rsyslog.com/e/2040 ] >> Jul 27 18:06:11 rsyslogd-2040: fatal error on disk queue 'action 0 >> queue[DA]', emergency switch to direct mode [v8.20.0 try >> http://www.rsyslog.com/e/2040 ] >> Jul 27 18:06:11 rsyslogd-2221: module 'imuxsock' already in this config, >> cannot be added [v8.20.0 try http://www.rsyslog.com/e/2221 ] >> Jul 27 18:06:11 rsyslogd-2221: module 'imklog' already in this config, >> cannot be added [v8.20.0 try http://www.rsyslog.com/e/2221 ] >> >> Thanks, >> >> Alec >> >> On Wed, Jul 27, 2016 at 11:03 AM, Rainer Gerhards < >> [email protected]> >> wrote: >> >> As I said, I tried for almost two weeks with different settings, for sure >>> more than 100 tries. This is just the latest state. In the end I focused >>> on >>> trusty, to save some time. If we can get that pic error go away, we can >>> look at the other ones. >>> >>> Rainer >>> >>> Sent from phone, thus brief. >>> >>> Am 27.07.2016 18:44 schrieb "David Lang" <[email protected]>: >>> >>> On Wed, 27 Jul 2016, Rainer Gerhards wrote: >>>> >>>> 2016-07-27 16:41 GMT+02:00 David Lang <[email protected]>: >>>> >>>>> >>>>> On Wed, 27 Jul 2016, Rainer Gerhards wrote: >>>>>> >>>>>> That's unfortunately not the case for the Ubuntu repo. I always >>>>>> >>>>>>> consistently recompiled everything. Actually the problem occurs >>>>>>> during >>>>>>> that >>>>>>> process, when e.g. liblognorm is compiled. >>>>>>> >>>>>>> >>>>>> >>>>>> I am only building for trusty, but I am using the rsyslog-pkg-ubuntu >>>>>> scripts >>>>>> to do the compiles, everything up to the point of uploading the >>>>>> results >>>>>> to >>>>>> the PPA >>>>>> >>>>>> are you getting the failure on trusty, or on other versions? >>>>>> >>>>>> >>>>> I get them on trusty and also on other versions. See here: >>>>> >>>>> https://launchpad.net/~adiscon/+archive/ubuntu/v8-devel/+packages >>>>> >>>>> I have the impression that the exact versions which fail change, but >>>>> trusty seems always affected. >>>>> >>>>> >>>> what I am seeing at that link is an error that it wasn't able to install >>>> libfastjson-dev (exact message: Depends: libfastjson-dev but it is not >>>> installable) >>>> >>>> hmm, digging further, it looks like the trusty amd64 build errors out >>>> >>> with >>> >>>> the -fPIC message, but all the other errors I see from the 22nd are >>>> uninstallable libfastjson-dev >>>> >>>> and indeed, there is not a libfastjson package for xenial or wily >>>> >>>> have you tried rebuilding libfastjson just in case something went wrong >>>> there? >>>> >>>> David Lang >>>> >>>> Rainer >>>> >>>>> >>>>> >>>>>> David Lang >>>>>> >>>>>> Rainer >>>>>> >>>>>>> >>>>>>> Sent from phone, thus brief. >>>>>>> >>>>>>> Am 27.07.2016 12:06 schrieb "David Lang" <[email protected]>: >>>>>>> >>>>>>> On Wed, 27 Jul 2016, Rainer Gerhards wrote: >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Thanks for the elaborate output. I think we may be getting closer. >>>>>>>> >>>>>>>>> This triggered my attention: >>>>>>>>> >>>>>>>>> --> Processing Dependency: libjson-c.so.2()(64bit) for package: >>>>>>>>> liblognorm1-1.1.3-1.el6.x86_64 >>>>>>>>> >>>>>>>>> I think we need to rebuild liblognorm with the new libfastjson >>>>>>>>> package. Otherwise we can see subtle problems with the json >>>>>>>>> >>>>>>>> handlers, >>> >>>> as there core structures are different (much more in 0.99.3 than >>>>>>>>> 0.99.2 of libfastjson). This doesn't explain yet the line number >>>>>>>>> problems we saw, but in anycase that's something to look at. I will >>>>>>>>> let you know when we have updates in the repo. >>>>>>>>> >>>>>>>>> >>>>>>>>> Yes, as I have been building with the git master for libfastjson >>>>>>>> and >>>>>>>> liblognorm for several months, I found that if I didn't recompile >>>>>>>> everything at once to be on the new version of libraries that both >>>>>>>> used, >>>>>>>> bad things happened. >>>>>>>> >>>>>>>> I've been suspecting that this sort of skew has been a large part of >>>>>>>> the >>>>>>>> repo problems of the last few weeks as new versions of libraries >>>>>>>> have >>>>>>>> been >>>>>>>> released at different times than rsyslog packages. I've been >>>>>>>> >>>>>>> suspecting >>> >>>> that if everything was recompiled, things would work (so 8.19.0 >>>>>>>> packages >>>>>>>> break with the new libraries, but I expected 8.20 packages compiled >>>>>>>> against >>>>>>>> the new libraries to then work) >>>>>>>> >>>>>>>> David Lang >>>>>>>> >>>>>>>> In parallel, I also install a CentOS6 (don't have RHEL) machine and >>>>>>>> >>>>>>>> >>>>>>>>> see what exactly it installes when we pull from the testing repo. >>>>>>>>> >>>>>>>>> Thanks again, >>>>>>>>> Rainer >>>>>>>>> >>>>>>>>> 2016-07-27 1:27 GMT+02:00 Alec Swan <[email protected]>: >>>>>>>>> >>>>>>>>> I configured /etc/yum.repos.d/rsyslog.repo to point to the >>>>>>>>> "testing" >>>>>>>>> >>>>>>>>>> rsyslog repo and removed all other yum repos. After that I >>>>>>>>>> >>>>>>>>> downloaded >>> >>>> all >>>>>>>>>> RPMS from the "testing" repo using yumdownloader and installed >>>>>>>>>> them >>>>>>>>>> with >>>>>>>>>> "rpm -ivh". The end result is the same. The following shows the >>>>>>>>>> output >>>>>>>>>> of >>>>>>>>>> the test. >>>>>>>>>> >>>>>>>>>> # more /etc/yum.repos.d/rsyslog.repo >>>>>>>>>> [rsyslog_v8] >>>>>>>>>> name=Adiscon CentOS-$releasever - local packages for $basearch >>>>>>>>>> baseurl= >>>>>>>>>> http://rpms.adiscon.com/testing/epel-$releasever/$basearch >>>>>>>>>> enabled=1 >>>>>>>>>> gpgcheck=0 >>>>>>>>>> gpgkey=http://rpms.adiscon.com/RPM-GPG-KEY-Adiscon >>>>>>>>>> protect=1 >>>>>>>>>> >>>>>>>>>> # yumdownloader --resolve rsyslog-mmnormalize-8.20.0 >>>>>>>>>> rsyslog-elasticsearch-8.20.0 rsyslog-mmutf8fix-8.20.0 >>>>>>>>>> Loaded plugins: fastestmirror >>>>>>>>>> Loading mirror speeds from cached hostfile >>>>>>>>>> ... >>>>>>>>>> rsyslog_v8 >>>>>>>>>> >>>>>>>>>> | 2.5 kB 00:00 >>>>>>>>>> rsyslog_v8/primary_db >>>>>>>>>> >>>>>>>>>> | 264 kB 00:00 >>>>>>>>>> --> Running transaction check >>>>>>>>>> ---> Package rsyslog-elasticsearch.x86_64 0:8.20.0-1.el6 will be >>>>>>>>>> installed >>>>>>>>>> --> Processing Dependency: rsyslog = 8.20.0-1.el6 for package: >>>>>>>>>> rsyslog-elasticsearch-8.20.0-1.el6.x86_64 >>>>>>>>>> ---> Package rsyslog-mmnormalize.x86_64 0:8.20.0-1.el6 will be >>>>>>>>>> installed >>>>>>>>>> --> Processing Dependency: liblognorm.so.2()(64bit) for package: >>>>>>>>>> rsyslog-mmnormalize-8.20.0-1.el6.x86_64 >>>>>>>>>> ---> Package rsyslog-mmutf8fix.x86_64 0:8.20.0-1.el6 will be >>>>>>>>>> installed >>>>>>>>>> --> Running transaction check >>>>>>>>>> ---> Package liblognorm1.x86_64 0:1.1.3-1.el6 will be installed >>>>>>>>>> --> Processing Dependency: libjson-c.so.2()(64bit) for package: >>>>>>>>>> liblognorm1-1.1.3-1.el6.x86_64 >>>>>>>>>> ---> Package rsyslog.x86_64 0:8.20.0-1.el6 will be installed >>>>>>>>>> --> Processing Dependency: libfastjson.so.4()(64bit) for package: >>>>>>>>>> rsyslog-8.20.0-1.el6.x86_64 >>>>>>>>>> --> Running transaction check >>>>>>>>>> ---> Package json-c.x86_64 0:0.11-11.el6 will be installed >>>>>>>>>> ---> Package libfastjson4.x86_64 0:0.99.3-2.el6 will be installed >>>>>>>>>> --> Finished Dependency Resolution >>>>>>>>>> rsyslog-mmnormalize-8.20.0-1.el6.x86_64.rpm >>>>>>>>>> >>>>>>>>>> | 13 kB 00:00 >>>>>>>>>> rsyslog-elasticsearch-8.20.0-1.el6.x86_64.rpm >>>>>>>>>> >>>>>>>>>> | 24 kB 00:00 >>>>>>>>>> rsyslog-mmutf8fix-8.20.0-1.el6.x86_64.rpm >>>>>>>>>> >>>>>>>>>> | 12 kB 00:00 >>>>>>>>>> rsyslog-8.20.0-1.el6.x86_64.rpm >>>>>>>>>> >>>>>>>>>> | 690 kB 00:00 >>>>>>>>>> json-c-0.11-11.el6.x86_64.rpm >>>>>>>>>> >>>>>>>>>> | 51 kB 00:00 >>>>>>>>>> libfastjson4-0.99.3-2.el6.x86_64.rpm >>>>>>>>>> >>>>>>>>>> | 46 kB 00:00 >>>>>>>>>> liblognorm1-1.1.3-1.el6.x86_64.rpm >>>>>>>>>> >>>>>>>>>> | 46 kB 00:00 >>>>>>>>>> >>>>>>>>>> # ls >>>>>>>>>> json-c-0.11-11.el6.x86_64.rpm >>>>>>>>>> liblognorm1-1.1.3-1.el6.x86_64.rpm >>>>>>>>>> rsyslog-elasticsearch-8.20.0-1.el6.x86_64.rpm >>>>>>>>>> rsyslog-mmutf8fix-8.20.0-1.el6.x86_64.rpm >>>>>>>>>> libfastjson4-0.99.3-2.el6.x86_64.rpm >>>>>>>>>> >>>>>>>>> rsyslog-8.20.0-1.el6.x86_64.rpm >>> >>>> rsyslog-mmnormalize-8.20.0-1.el6.x86_64.rpm >>>>>>>>>> >>>>>>>>>> # rpm -ivh *.rpm >>>>>>>>>> warning: libfastjson4-0.99.3-2.el6.x86_64.rpm: Header V3 RSA/SHA1 >>>>>>>>>> Signature, key ID e00b8985: NOKEY >>>>>>>>>> Preparing... >>>>>>>>>> ########################################### >>>>>>>>>> [100%] >>>>>>>>>> 1:libfastjson4 >>>>>>>>>> ########################################### >>>>>>>>>> [ >>>>>>>>>> 14%] >>>>>>>>>> 2:rsyslog >>>>>>>>>> ########################################### >>>>>>>>>> [ >>>>>>>>>> 29%] >>>>>>>>>> 3:json-c >>>>>>>>>> ########################################### >>>>>>>>>> [ >>>>>>>>>> 43%] >>>>>>>>>> 4:liblognorm1 >>>>>>>>>> ########################################### >>>>>>>>>> [ >>>>>>>>>> 57%] >>>>>>>>>> 5:rsyslog-mmnormalize >>>>>>>>>> ########################################### >>>>>>>>>> [ >>>>>>>>>> 71%] >>>>>>>>>> 6:rsyslog-elasticsearch >>>>>>>>>> ########################################### >>>>>>>>>> [ >>>>>>>>>> 86%] >>>>>>>>>> 7:rsyslog-mmutf8fix >>>>>>>>>> ########################################### >>>>>>>>>> [100%] >>>>>>>>>> >>>>>>>>>> # service rsyslog start >>>>>>>>>> Starting system logger: [ OK >>>>>>>>>> ] >>>>>>>>>> >>>>>>>>>> # ps -ax | grep rsys >>>>>>>>>> Warning: bad syntax, perhaps a bogus '-'? See >>>>>>>>>> /usr/share/doc/procps-3.2.8/FAQ >>>>>>>>>> 1020 pts/0 S+ 0:00 grep rsys >>>>>>>>>> >>>>>>>>>> # dmesg >>>>>>>>>> ... >>>>>>>>>> rs:main Q:Reg[1009]: segfault at 0 ip 00007f79a5e4d2b6 sp >>>>>>>>>> 00007f799e66f7e8 >>>>>>>>>> error 4 in libc-2.12.so[7f79a5d25000+18a000] >>>>>>>>>> >>>>>>>>>> # service rsyslog start >>>>>>>>>> Starting system logger: [ OK >>>>>>>>>> ] >>>>>>>>>> >>>>>>>>>> # dmesg >>>>>>>>>> ... >>>>>>>>>> rs:main Q:Reg[1009]: segfault at 0 ip 00007f79a5e4d2b6 sp >>>>>>>>>> 00007f799e66f7e8 >>>>>>>>>> error 4 in libc-2.12.so[7f79a5d25000+18a000] >>>>>>>>>> rs:main Q:Reg[1063]: segfault at 1b ip 00007f9a44249778 sp >>>>>>>>>> 00007f9a3c1de878 >>>>>>>>>> error 6 in libfastjson.so.4.0.0[7f9a44246000+9000] >>>>>>>>>> >>>>>>>>>> # service rsyslog start >>>>>>>>>> Starting system logger: [ OK >>>>>>>>>> ] >>>>>>>>>> >>>>>>>>>> # dmesg >>>>>>>>>> ... >>>>>>>>>> rs:main Q:Reg[1009]: segfault at 0 ip 00007f79a5e4d2b6 sp >>>>>>>>>> 00007f799e66f7e8 >>>>>>>>>> error 4 in libc-2.12.so[7f79a5d25000+18a000] >>>>>>>>>> rs:main Q:Reg[1063]: segfault at 1b ip 00007f9a44249778 sp >>>>>>>>>> 00007f9a3c1de878 >>>>>>>>>> error 6 in libfastjson.so.4.0.0[7f9a44246000+9000] >>>>>>>>>> rs:main Q:Reg[1093]: segfault at 0 ip 00007f32e00c22b6 sp >>>>>>>>>> 00007f32d88e47e8 >>>>>>>>>> error 4 in libc-2.12.so[7f32dff9a000+18a000] >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> Alec >>>>>>>>>> >>>>>>>>>> On Tue, Jul 26, 2016 at 10:18 AM, Rainer Gerhards < >>>>>>>>>> [email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Am 26.07.2016 17:58 schrieb "Alec Swan" <[email protected]>: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I downloaded the packages from adiscon "testing" repo, then >>>>>>>>>>>> uploaded >>>>>>>>>>>> them >>>>>>>>>>>> to our local Nexus repo and then installed them with yum. So, >>>>>>>>>>>> the >>>>>>>>>>>> source >>>>>>>>>>>> >>>>>>>>>>>> is >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> our local Nexus repo. >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Unfortunately, the "testing" repo does not seem to have libee.so >>>>>>>>>>>> >>>>>>>>>>>> dependency >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> required by libfastjson, so I can't just use "testing" repo to >>>>>>>>>>> >>>>>>>>>>>> install >>>>>>>>>>>> >>>>>>>>>>>> all >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> rsyslog dependencies. >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Something is fishy here. I just checked the spec file, and >>>>>>>>>>> libfastjson >>>>>>>>>>> does >>>>>>>>>>> not have a dependency on libee. Is there a problem with the >>>>>>>>>>> >>>>>>>>>> staging >>> >>>> process? Or can you try from the original repo, not the mirror? >>>>>>>>>>> >>>>>>>>>>> Rainer >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> >>>>>>>>>>>> Alec >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Jul 26, 2016 at 9:36 AM, Rainer Gerhards < >>>>>>>>>>>> >>>>>>>>>>>> [email protected] >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> 2016-07-26 17:35 GMT+02:00 Alec Swan <[email protected]>: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Rainer, per Florian's request in "experimental rpms for >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> rsyslog-8.20.0" I >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> downloaded RPMs from the "testing" repo baseurl= >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> http://rpms.adiscon.com/testing/epel-$releasever/$basearch >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> Can you check where the system says it actually installed them >>>>>>>>>>>>> from. >>>>>>>>>>>>> The line numbers do not match the RPMs Florian created, thus I >>>>>>>>>>>>> ask. >>>>>>>>>>>>> >>>>>>>>>>>>> Rainer >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Alec >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Tue, Jul 26, 2016 at 7:30 AM, Rainer Gerhards < >>>>>>>>>>>>>> >>>>>>>>>>>>>> [email protected]> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> I today tried quite a while to reproduce the issue with your >>>>>>>>>>>>>> config, >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> unfortunately without any success. For now, I conclude that >>>>>>>>>>>>>>> >>>>>>>>>>>>>> this >>> >>>> is >>>>>>>>>>>>>>> not a general problem, but one induced by your environment >>>>>>>>>>>>>>> and >>>>>>>>>>>>>>> data. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I have then looked again at the traces and the few line >>>>>>>>>>>>>>> >>>>>>>>>>>>>> numbers >>> >>>> it >>>>>>>>>>>>>>> contains. Now comes the interesting fact. I notice that there >>>>>>>>>>>>>>> is a >>>>>>>>>>>>>>> mismatch between the line numbers shown and the code that >>>>>>>>>>>>>>> actually >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> was >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>> used to build the packages (judging from git and the source >>>>>>>>>>> RPMs). >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>>> So >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>> I wonder what might have gone wrong here. Can you please let me >>>>>>>>>>> >>>>>>>>>> know >>> >>>> >>>>>>>>>>>> >>>>>>>>>>>>> which exact repro you installed from and potentially which >>>>>>>>>>>>>>> >>>>>>>>>>>>>> other >>> >>>> >>>>>>>>>>>>>>> repos >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>> you have configured on that machine. It might also be useful to >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>>> query >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>> the system from where exactly the packages have ben installed. I >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>>> found >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>> [1], which hopefully helps to get that information. >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>>> Once I have this information, I can check if there are any >>>>>>>>>>>>>>> version >>>>>>>>>>>>>>> quirks (note that commit [2] fixed a situation that could >>>>>>>>>>>>>>> >>>>>>>>>>>>>> cause >>> >>>> exactly what you see). If there is no issue with the package >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> sources, >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>> I unfortunately need your help in further troubleshooting this in >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>>> your >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>> environment. But let's first see what the repo check will bring >>>>>>>>>>> >>>>>>>>>> up. >>> >>>> >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>>> Rainer >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [1] >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>> http://serverfault.com/questions/62026/how-to-know-from-which-yum-repository-a-package-has-been-installed >>> >>>> >>>>>>>>>>> [2] >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>> https://github.com/rsyslog/libfastjson/commit/0499ec8df09c913156ead426c171031b566dbe7d >>> >>>> >>>>>>>>>>> >>>>>>>>>>> 2016-07-23 19:28 GMT+02:00 Alec Swan <[email protected]>: >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>>> Yes, that should be OK. Is there a link to the build >>>>>>>>>>>>>>> server/CI >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> env >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>> can >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> use? >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Alec >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Sat, Jul 23, 2016 at 7:08 AM, Rainer Gerhards < >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> [email protected]> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks. Would it be OK for your if a somewhat mangled >>>>>>>>>>>>>>>> version >>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> these >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>> configs show up inside the testbench? I think about adding a >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> specific >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>> test for it, that would be great for the future as well. I'd >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>>>>> probably >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>> try to strip it a bit down (e.g. no ES but writing to file, which >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> would enable runs on all CI environments and when no ES is >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> available >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>> -- I assume this would also trigger the bug). >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Please let me know, >>>>>>>>>>>>>>>>> Rainer >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 2016-07-22 17:46 GMT+02:00 Alec Swan <[email protected]>: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Rainer, I attached /etc/rsyslog.conf and >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> /etc/rsyslog.d/rsyslog-custom.conf >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Alec >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Fri, Jul 22, 2016 at 12:42 AM, Rainer Gerhards < >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> [email protected]> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> unfortunately, we do not seem to have debug symbols, so I >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> don't >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>> see >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> much more. Can you share your full config (including all >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> includes, if >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>> any)? Maybe I can get it to abort in my lab as well. >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Rainer >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> 2016-07-21 22:02 GMT+02:00 Alec Swan <[email protected] >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> : >>>> >>>>> >>>>>>>>>>>>>>>>>>> Rainer, here is the output from running rsyslog under >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> valgrind: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> [root valgrind-3.11.0]# valgrind rsyslogd >>>>>>>>>>>> >>>>>>>>>>>>> ==2371== Memcheck, a memory error detector >>>>>>>>>>>>>>>>>>>> ==2371== Copyright (C) 2002-2015, and GNU GPL'd, by >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Julian >>> >>>> >>>>>>>>>>>>>>>>>>>> Seward >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> et >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> al. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> ==2371== Using Valgrind-3.11.0 and LibVEX; rerun with -h >>>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> copyright >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> info >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> ==2371== Command: rsyslogd >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> ==2371== >>>>>>>>>>>>>>>>>>>> ==2371== Conditional jump or move depends on >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> uninitialised >>> >>>> >>>>>>>>>>>>>>>>>>>> value(s) >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> ==2371== at 0x16A414: cnfexprDestruct (in /sbin/rsyslogd) >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> ==2371== by 0x16B36D: cnfstmtOptimize (in >>>>>>>>>>>>>>>>>>>> /sbin/rsyslogd) >>>>>>>>>>>>>>>>>>>> ==2371== by 0x146317: ??? (in /sbin/rsyslogd) >>>>>>>>>>>>>>>>>>>> ==2371== by 0x12E5BE: llExecFunc (in /sbin/rsyslogd) >>>>>>>>>>>>>>>>>>>> ==2371== by 0x146176: rulesetOptimizeAll (in >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> /sbin/rsyslogd) >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>> ==2371== by 0x123EF6: ??? (in /sbin/rsyslogd) >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> ==2371== by 0x158A68: ??? (in /sbin/rsyslogd) >>>>>>>>>>>>>>>>>>>> ==2371== by 0x159008: main (in /sbin/rsyslogd) >>>>>>>>>>>>>>>>>>>> ==2371== >>>>>>>>>>>>>>>>>>>> ==2375== Warning: invalid file descriptor 99990 in >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> syscall >>> >>>> >>>>>>>>>>>>>>>>>>>> close() >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> ==2371== >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> ==2371== HEAP SUMMARY: >>>>>>>>>>>>>>>>>>>> ==2371== in use at exit: 209,122 bytes in 2,121 >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> blocks >>> >>>> ==2371== total heap usage: 11,004 allocs, 8,883 frees, >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> 4,319,525 >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> bytes >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> allocated >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> ==2371== >>>>>>>>>>>>>>>>>>>> ==2371== LEAK SUMMARY: >>>>>>>>>>>>>>>>>>>> ==2371== definitely lost: 26 bytes in 1 blocks >>>>>>>>>>>>>>>>>>>> ==2371== indirectly lost: 0 bytes in 0 blocks >>>>>>>>>>>>>>>>>>>> ==2371== possibly lost: 88 bytes in 2 blocks >>>>>>>>>>>>>>>>>>>> ==2371== still reachable: 209,008 bytes in 2,118 >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> blocks >>> >>>> ==2371== suppressed: 0 bytes in 0 blocks >>>>>>>>>>>>>>>>>>>> ==2371== Rerun with --leak-check=full to see details of >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> leaked >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>> memory >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>>> ==2371== >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> ==2371== For counts of detected and suppressed errors, >>>>>>>>>>>>>>>>>>>> rerun >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> with: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> -v >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> ==2371== Use --track-origins=yes to see where uninitialised >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> values >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> come >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> from >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> ==2371== ERROR SUMMARY: 1 errors from 1 contexts >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> (suppressed: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>> 114 >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> from 7) >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [root valgrind-3.11.0]# ==2375== Thread 6 rs:main Q:Reg: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> ==2375== Invalid read of size 8 >>>>>>>>>>>>>>>>>>>> ==2375== at 0x56607BE: fjson_object_iter_next >>>>>>>>>>>>>>>>>>>> (json_object_iterator.c:119) >>>>>>>>>>>>>>>>>>>> ==2375== by 0x5660863: fjson_object_iter_begin >>>>>>>>>>>>>>>>>>>> (json_object_iterator.c:77) >>>>>>>>>>>>>>>>>>>> ==2375== by 0x126F09: ??? (in /sbin/rsyslogd) >>>>>>>>>>>>>>>>>>>> ==2375== by 0x1270A0: msgAddJSON (in /sbin/rsyslogd) >>>>>>>>>>>>>>>>>>>> ==2375== by 0x7242F2D: ??? (in >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> /lib64/rsyslog/mmnormalize.so) >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>> ==2375== by 0x14EA64: ??? (in /sbin/rsyslogd) >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> ==2375== by 0x14F561: ??? (in /sbin/rsyslogd) >>>>>>>>>>>>>>>>>>>> ==2375== by 0x14F923: ??? (in /sbin/rsyslogd) >>>>>>>>>>>>>>>>>>>> ==2375== by 0x146AB6: ??? (in /sbin/rsyslogd) >>>>>>>>>>>>>>>>>>>> ==2375== by 0x1473DA: ??? (in /sbin/rsyslogd) >>>>>>>>>>>>>>>>>>>> ==2375== by 0x156FE3: ??? (in /sbin/rsyslogd) >>>>>>>>>>>>>>>>>>>> ==2375== by 0x1457D8: ??? (in /sbin/rsyslogd) >>>>>>>>>>>>>>>>>>>> ==2375== Address 0xf57d338 is 0 bytes after a block of >>>>>>>>>>>>>>>>>>>> size >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> 72 >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>> alloc'd >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>>>>> ==2375== at 0x4A05FEF: calloc (vg_replace_malloc.c:711) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> ==2375== by 0x7659871: json_object_new >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> (json_object.c:178) >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>> ==2375== by 0x765A61A: json_object_new_object >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> (json_object.c:354) >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> ==2375== by 0x744A355: ln_normalize (in >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> /usr/lib64/liblognorm.so.2.0.0) >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> ==2375== by 0x7242EEB: ??? (in >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> /lib64/rsyslog/mmnormalize.so) >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>> ==2375== by 0x14EA64: ??? (in /sbin/rsyslogd) >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> ==2375== by 0x14F561: ??? (in /sbin/rsyslogd) >>>>>>>>>>>>>>>>>>>> ==2375== by 0x14F923: ??? (in /sbin/rsyslogd) >>>>>>>>>>>>>>>>>>>> ==2375== by 0x146AB6: ??? (in /sbin/rsyslogd) >>>>>>>>>>>>>>>>>>>> ==2375== by 0x1473DA: ??? (in /sbin/rsyslogd) >>>>>>>>>>>>>>>>>>>> ==2375== by 0x156FE3: ??? (in /sbin/rsyslogd) >>>>>>>>>>>>>>>>>>>> ==2375== by 0x1457D8: ??? (in /sbin/rsyslogd) >>>>>>>>>>>>>>>>>>>> ==2375== >>>>>>>>>>>>>>>>>>>> ==2375== Invalid read of size 4 >>>>>>>>>>>>>>>>>>>> ==2375== at 0x5660778: fjson_object_get >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> (json_object.c:176) >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>> ==2375== by 0x126F57: ??? (in /sbin/rsyslogd) >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> ==2375== by 0x1270A0: msgAddJSON (in /sbin/rsyslogd) >>>>>>>>>>>>>>>>>>>> ==2375== by 0x7242F2D: ??? (in >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> /lib64/rsyslog/mmnormalize.so) >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>> ==2375== by 0x14EA64: ??? (in /sbin/rsyslogd) >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> ==2375== by 0x14F561: ??? (in /sbin/rsyslogd) >>>>>>>>>>>>>>>>>>>> ==2375== by 0x14F923: ??? (in /sbin/rsyslogd) >>>>>>>>>>>>>>>>>>>> ==2375== by 0x146AB6: ??? (in /sbin/rsyslogd) >>>>>>>>>>>>>>>>>>>> ==2375== by 0x1473DA: ??? (in /sbin/rsyslogd) >>>>>>>>>>>>>>>>>>>> ==2375== by 0x156FE3: ??? (in /sbin/rsyslogd) >>>>>>>>>>>>>>>>>>>> ==2375== by 0x1457D8: ??? (in /sbin/rsyslogd) >>>>>>>>>>>>>>>>>>>> ==2375== by 0x13FB5D: wtiWorker (in /sbin/rsyslogd) >>>>>>>>>>>>>>>>>>>> ==2375== Address 0x60000001e is not stack'd, malloc'd >>>>>>>>>>>>>>>>>>>> or >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> (recently) >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> free'd >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> ==2375== >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> ==2375== >>>>>>>>>>>>>>>>>>>> ==2375== Process terminating with default action of >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> signal >>> >>>> >>>>>>>>>>>>>>>>>>>> 11 >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>> (SIGSEGV) >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>>>>> ==2375== Access not within mapped region at address >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> 0x60000001E >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>> ==2375== at 0x5660778: fjson_object_get >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> (json_object.c:176) >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>> ==2375== by 0x126F57: ??? (in /sbin/rsyslogd) >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> ==2375== by 0x1270A0: msgAddJSON (in /sbin/rsyslogd) >>>>>>>>>>>>>>>>>>>> ==2375== by 0x7242F2D: ??? (in >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> /lib64/rsyslog/mmnormalize.so) >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>> ==2375== by 0x14EA64: ??? (in /sbin/rsyslogd) >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> ==2375== by 0x14F561: ??? (in /sbin/rsyslogd) >>>>>>>>>>>>>>>>>>>> ==2375== by 0x14F923: ??? (in /sbin/rsyslogd) >>>>>>>>>>>>>>>>>>>> ==2375== by 0x146AB6: ??? (in /sbin/rsyslogd) >>>>>>>>>>>>>>>>>>>> ==2375== by 0x1473DA: ??? (in /sbin/rsyslogd) >>>>>>>>>>>>>>>>>>>> ==2375== by 0x156FE3: ??? (in /sbin/rsyslogd) >>>>>>>>>>>>>>>>>>>> ==2375== by 0x1457D8: ??? (in /sbin/rsyslogd) >>>>>>>>>>>>>>>>>>>> ==2375== by 0x13FB5D: wtiWorker (in /sbin/rsyslogd) >>>>>>>>>>>>>>>>>>>> ==2375== If you believe this happened as a result of a >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> stack >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>> ==2375== overflow in your program's main thread (unlikely >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> but >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>> ==2375== possible), you can try to increase the size of the >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> ==2375== main thread stack using the --main-stacksize= >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> flag. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>> ==2375== The main thread stack size used in this run was >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> 10485760. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> ==2375== >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> ==2375== HEAP SUMMARY: >>>>>>>>>>>>>>>>>>>> ==2375== in use at exit: 2,611,014 bytes in 11,732 >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> blocks >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>> ==2375== total heap usage: 27,013 allocs, 15,281 frees, >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> 8,524,299 >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>> bytes >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> allocated >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> ==2375== >>>>>>>>>>>>>>>>>>>> ==2375== LEAK SUMMARY: >>>>>>>>>>>>>>>>>>>> ==2375== definitely lost: 4,117 bytes in 11 blocks >>>>>>>>>>>>>>>>>>>> ==2375== indirectly lost: 24,922 bytes in 20 blocks >>>>>>>>>>>>>>>>>>>> ==2375== possibly lost: 4,464 bytes in 12 blocks >>>>>>>>>>>>>>>>>>>> ==2375== still reachable: 2,577,511 bytes in 11,689 >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> blocks >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
