Re: [rsyslog] libfastjson 0.99.3 released

David Lang Wed, 27 Jul 2016 07:42:06 -0700

On Wed, 27 Jul 2016, Rainer Gerhards wrote:

That's unfortunately not the case for the Ubuntu repo. I always
consistently recompiled everything. Actually the problem occurs during that
process, when e.g. liblognorm is compiled.

I am only building for trusty, but I am using the rsyslog-pkg-ubuntu scripts todo the compiles, everything up to the point of uploading the results to the PPA


are you getting the failure on trusty, or on other versions?

David Lang

Rainer

Sent from phone, thus brief.

Am 27.07.2016 12:06 schrieb "David Lang" <[email protected]>:

On Wed, 27 Jul 2016, Rainer Gerhards wrote:

Thanks for the elaborate output. I think we may be getting closer.
This triggered my attention:

--> Processing Dependency: libjson-c.so.2()(64bit) for package:
liblognorm1-1.1.3-1.el6.x86_64

I think we need to rebuild liblognorm with the new libfastjson
package. Otherwise we can see subtle problems with the json handlers,
as there core structures are different (much more in 0.99.3 than
0.99.2 of libfastjson). This doesn't explain yet the line number
problems we saw, but in anycase that's something to look at. I will
let you know when we have updates in the repo.


Yes, as I have been building with the git master for libfastjson and
liblognorm for several months, I found that if I didn't recompile
everything at once to be on the new version of libraries that both used,
bad things happened.

I've been suspecting that this sort of skew has been a large part of the
repo problems of the last few weeks as new versions of libraries have been
released at different times than rsyslog packages. I've been suspecting
that if everything was recompiled, things would work (so 8.19.0 packages
break with the new libraries, but I expected 8.20 packages compiled against
the new libraries to then work)

David Lang

In parallel, I also install a CentOS6 (don't have RHEL) machine and

see what exactly it installes when we pull from the testing repo.

Thanks again,
Rainer

2016-07-27 1:27 GMT+02:00 Alec Swan <[email protected]>:

I configured /etc/yum.repos.d/rsyslog.repo to point to the "testing"
rsyslog repo and removed all other yum repos. After that I downloaded all
RPMS from the "testing" repo using yumdownloader and installed them with
"rpm -ivh". The end result is the same. The following shows the output of
the test.

# more /etc/yum.repos.d/rsyslog.repo
[rsyslog_v8]
name=Adiscon CentOS-$releasever - local packages for $basearch
baseurl=http://rpms.adiscon.com/testing/epel-$releasever/$basearch
enabled=1
gpgcheck=0
gpgkey=http://rpms.adiscon.com/RPM-GPG-KEY-Adiscon
protect=1

# yumdownloader --resolve rsyslog-mmnormalize-8.20.0
rsyslog-elasticsearch-8.20.0 rsyslog-mmutf8fix-8.20.0
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
...
rsyslog_v8

  | 2.5 kB     00:00
rsyslog_v8/primary_db

 | 264 kB     00:00
--> Running transaction check
---> Package rsyslog-elasticsearch.x86_64 0:8.20.0-1.el6 will be
installed
--> Processing Dependency: rsyslog = 8.20.0-1.el6 for package:
rsyslog-elasticsearch-8.20.0-1.el6.x86_64
---> Package rsyslog-mmnormalize.x86_64 0:8.20.0-1.el6 will be installed
--> Processing Dependency: liblognorm.so.2()(64bit) for package:
rsyslog-mmnormalize-8.20.0-1.el6.x86_64
---> Package rsyslog-mmutf8fix.x86_64 0:8.20.0-1.el6 will be installed
--> Running transaction check
---> Package liblognorm1.x86_64 0:1.1.3-1.el6 will be installed
--> Processing Dependency: libjson-c.so.2()(64bit) for package:
liblognorm1-1.1.3-1.el6.x86_64
---> Package rsyslog.x86_64 0:8.20.0-1.el6 will be installed
--> Processing Dependency: libfastjson.so.4()(64bit) for package:
rsyslog-8.20.0-1.el6.x86_64
--> Running transaction check
---> Package json-c.x86_64 0:0.11-11.el6 will be installed
---> Package libfastjson4.x86_64 0:0.99.3-2.el6 will be installed
--> Finished Dependency Resolution
rsyslog-mmnormalize-8.20.0-1.el6.x86_64.rpm

 |  13 kB     00:00
rsyslog-elasticsearch-8.20.0-1.el6.x86_64.rpm

 |  24 kB     00:00
rsyslog-mmutf8fix-8.20.0-1.el6.x86_64.rpm

 |  12 kB     00:00
rsyslog-8.20.0-1.el6.x86_64.rpm

 | 690 kB     00:00
json-c-0.11-11.el6.x86_64.rpm

 |  51 kB     00:00
libfastjson4-0.99.3-2.el6.x86_64.rpm

  |  46 kB     00:00
liblognorm1-1.1.3-1.el6.x86_64.rpm

  |  46 kB     00:00

# ls
json-c-0.11-11.el6.x86_64.rpm         liblognorm1-1.1.3-1.el6.x86_64.rpm
 rsyslog-elasticsearch-8.20.0-1.el6.x86_64.rpm
 rsyslog-mmutf8fix-8.20.0-1.el6.x86_64.rpm
libfastjson4-0.99.3-2.el6.x86_64.rpm  rsyslog-8.20.0-1.el6.x86_64.rpm
rsyslog-mmnormalize-8.20.0-1.el6.x86_64.rpm

# rpm -ivh *.rpm
warning: libfastjson4-0.99.3-2.el6.x86_64.rpm: Header V3 RSA/SHA1
Signature, key ID e00b8985: NOKEY
Preparing...                ###########################################
[100%]
   1:libfastjson4           ########################################### [
14%]
   2:rsyslog                ########################################### [
29%]
   3:json-c                 ########################################### [
43%]
   4:liblognorm1            ########################################### [
57%]
   5:rsyslog-mmnormalize    ########################################### [
71%]
   6:rsyslog-elasticsearch  ########################################### [
86%]
   7:rsyslog-mmutf8fix      ###########################################
[100%]

# service rsyslog start
Starting system logger:                                    [  OK  ]

# ps -ax | grep rsys
Warning: bad syntax, perhaps a bogus '-'? See
/usr/share/doc/procps-3.2.8/FAQ
 1020 pts/0    S+     0:00 grep rsys

# dmesg
...
rs:main Q:Reg[1009]: segfault at 0 ip 00007f79a5e4d2b6 sp
00007f799e66f7e8
error 4 in libc-2.12.so[7f79a5d25000+18a000]

# service rsyslog start
Starting system logger:                                    [  OK  ]

# dmesg
...
rs:main Q:Reg[1009]: segfault at 0 ip 00007f79a5e4d2b6 sp
00007f799e66f7e8
error 4 in libc-2.12.so[7f79a5d25000+18a000]
rs:main Q:Reg[1063]: segfault at 1b ip 00007f9a44249778 sp
00007f9a3c1de878
error 6 in libfastjson.so.4.0.0[7f9a44246000+9000]

# service rsyslog start
Starting system logger:                                    [  OK  ]

# dmesg
...
rs:main Q:Reg[1009]: segfault at 0 ip 00007f79a5e4d2b6 sp
00007f799e66f7e8
error 4 in libc-2.12.so[7f79a5d25000+18a000]
rs:main Q:Reg[1063]: segfault at 1b ip 00007f9a44249778 sp
00007f9a3c1de878
error 6 in libfastjson.so.4.0.0[7f9a44246000+9000]
rs:main Q:Reg[1093]: segfault at 0 ip 00007f32e00c22b6 sp
00007f32d88e47e8
error 4 in libc-2.12.so[7f32dff9a000+18a000]


Thanks,

Alec

On Tue, Jul 26, 2016 at 10:18 AM, Rainer Gerhards <
[email protected]>
wrote:

Am 26.07.2016 17:58 schrieb "Alec Swan" <[email protected]>:


I downloaded the packages from adiscon "testing" repo, then uploaded
them
to our local Nexus repo and then installed them with yum. So, the
source

is

our local Nexus repo.

Unfortunately, the "testing" repo does not seem to have libee.so

dependency

required by libfastjson, so I can't just use "testing" repo to install

all

rsyslog dependencies.


Something is fishy here. I just checked the spec file, and libfastjson
does
not have a dependency on libee. Is there a problem with the staging
process? Or can you try from the original repo, not the mirror?

Rainer



Thanks,

Alec

On Tue, Jul 26, 2016 at 9:36 AM, Rainer Gerhards <

[email protected]


wrote:

2016-07-26 17:35 GMT+02:00 Alec Swan <[email protected]>:

Rainer, per Florian's request in "experimental rpms for

rsyslog-8.20.0" I

downloaded RPMs from the "testing" repo baseurl=

http://rpms.adiscon.com/testing/epel-$releasever/$basearch



Can you check where the system says it actually installed them from.
The line numbers do not match the RPMs Florian created, thus I ask.

Rainer


Thanks,

Alec

On Tue, Jul 26, 2016 at 7:30 AM, Rainer Gerhards <

[email protected]>

wrote:

I today tried quite a while to reproduce the issue with your config,

unfortunately without any success. For now, I conclude that this is
not a general problem, but one induced by your environment and data.

I have then looked again at the traces and the few line numbers it
contains. Now comes the interesting fact. I notice that there is a
mismatch between the line numbers shown and the code that actually

was

used to build the packages (judging from git and the source RPMs).

So

I wonder what might have gone wrong here. Can you please let me know

which exact repro you installed from and potentially which other

repos

you have configured on that machine. It might also be useful to

query

the system from where exactly the packages have ben installed. I

found

[1], which hopefully helps to get that information.


Once I have this information, I can check if there are any version
quirks (note that commit [2] fixed a situation that could cause
exactly what you see). If there is no issue with the package

sources,

I unfortunately need your help in further troubleshooting this in

your

environment. But let's first see what the repo check will bring up.


Rainer

[1]


http://serverfault.com/questions/62026/how-to-know-from-which-yum-repository-a-package-has-been-installed

[2]


https://github.com/rsyslog/libfastjson/commit/0499ec8df09c913156ead426c171031b566dbe7d

2016-07-23 19:28 GMT+02:00 Alec Swan <[email protected]>:

Yes, that should be OK. Is there a link to the build server/CI env

can

use?


Thanks,

Alec

On Sat, Jul 23, 2016 at 7:08 AM, Rainer Gerhards <

[email protected]>

wrote:

Thanks. Would it be OK for your if a somewhat mangled version of

these

configs show up inside the testbench? I think about adding a

specific

test for it, that would be great for the future as well. I'd

probably

try to strip it a bit down (e.g. no ES but writing to file, which

would enable runs on all CI environments and when no ES is

available

-- I assume this would also trigger the bug).


Please let me know,
Rainer

2016-07-22 17:46 GMT+02:00 Alec Swan <[email protected]>:

Rainer, I attached /etc/rsyslog.conf and

/etc/rsyslog.d/rsyslog-custom.conf


Thanks,

Alec

On Fri, Jul 22, 2016 at 12:42 AM, Rainer Gerhards <

[email protected]>

wrote:

unfortunately, we do not seem to have debug symbols, so I

don't

see

much more. Can you share your full config (including all

includes, if

any)? Maybe I can get it to abort in my lab as well.


Rainer

2016-07-21 22:02 GMT+02:00 Alec Swan <[email protected]>:

Rainer, here is the output from running rsyslog under

valgrind:

[root valgrind-3.11.0]# valgrind rsyslogd
==2371== Memcheck, a memory error detector
==2371== Copyright (C) 2002-2015, and GNU GPL'd, by Julian

Seward

et

al.

==2371== Using Valgrind-3.11.0 and LibVEX; rerun with -h for

copyright

info

==2371== Command: rsyslogd
==2371==
==2371== Conditional jump or move depends on uninitialised

value(s)

==2371==    at 0x16A414: cnfexprDestruct (in /sbin/rsyslogd)

==2371==    by 0x16B36D: cnfstmtOptimize (in /sbin/rsyslogd)
==2371==    by 0x146317: ??? (in /sbin/rsyslogd)
==2371==    by 0x12E5BE: llExecFunc (in /sbin/rsyslogd)
==2371==    by 0x146176: rulesetOptimizeAll (in

/sbin/rsyslogd)

==2371==    by 0x123EF6: ??? (in /sbin/rsyslogd)

==2371==    by 0x158A68: ??? (in /sbin/rsyslogd)
==2371==    by 0x159008: main (in /sbin/rsyslogd)
==2371==
==2375== Warning: invalid file descriptor 99990 in syscall

close()

==2371==

==2371== HEAP SUMMARY:
==2371==     in use at exit: 209,122 bytes in 2,121 blocks
==2371==   total heap usage: 11,004 allocs, 8,883 frees,

4,319,525

bytes

allocated

==2371==
==2371== LEAK SUMMARY:
==2371==    definitely lost: 26 bytes in 1 blocks
==2371==    indirectly lost: 0 bytes in 0 blocks
==2371==      possibly lost: 88 bytes in 2 blocks
==2371==    still reachable: 209,008 bytes in 2,118 blocks
==2371==         suppressed: 0 bytes in 0 blocks
==2371== Rerun with --leak-check=full to see details of

leaked

memory

==2371==

==2371== For counts of detected and suppressed errors, rerun

with:

-v

==2371== Use --track-origins=yes to see where uninitialised

values

come

from

==2371== ERROR SUMMARY: 1 errors from 1 contexts

(suppressed:

from 7)

[root valgrind-3.11.0]# ==2375== Thread 6 rs:main Q:Reg:

==2375== Invalid read of size 8
==2375==    at 0x56607BE: fjson_object_iter_next
(json_object_iterator.c:119)
==2375==    by 0x5660863: fjson_object_iter_begin
(json_object_iterator.c:77)
==2375==    by 0x126F09: ??? (in /sbin/rsyslogd)
==2375==    by 0x1270A0: msgAddJSON (in /sbin/rsyslogd)
==2375==    by 0x7242F2D: ??? (in

/lib64/rsyslog/mmnormalize.so)

==2375==    by 0x14EA64: ??? (in /sbin/rsyslogd)

==2375==    by 0x14F561: ??? (in /sbin/rsyslogd)
==2375==    by 0x14F923: ??? (in /sbin/rsyslogd)
==2375==    by 0x146AB6: ??? (in /sbin/rsyslogd)
==2375==    by 0x1473DA: ??? (in /sbin/rsyslogd)
==2375==    by 0x156FE3: ??? (in /sbin/rsyslogd)
==2375==    by 0x1457D8: ??? (in /sbin/rsyslogd)
==2375==  Address 0xf57d338 is 0 bytes after a block of size

alloc'd

==2375==    at 0x4A05FEF: calloc (vg_replace_malloc.c:711)

==2375==    by 0x7659871: json_object_new

(json_object.c:178)

==2375==    by 0x765A61A: json_object_new_object

(json_object.c:354)

==2375==    by 0x744A355: ln_normalize (in

/usr/lib64/liblognorm.so.2.0.0)

==2375==    by 0x7242EEB: ??? (in

/lib64/rsyslog/mmnormalize.so)

==2375==    by 0x14EA64: ??? (in /sbin/rsyslogd)

==2375==    by 0x14F561: ??? (in /sbin/rsyslogd)
==2375==    by 0x14F923: ??? (in /sbin/rsyslogd)
==2375==    by 0x146AB6: ??? (in /sbin/rsyslogd)
==2375==    by 0x1473DA: ??? (in /sbin/rsyslogd)
==2375==    by 0x156FE3: ??? (in /sbin/rsyslogd)
==2375==    by 0x1457D8: ??? (in /sbin/rsyslogd)
==2375==
==2375== Invalid read of size 4
==2375==    at 0x5660778: fjson_object_get

(json_object.c:176)

==2375==    by 0x126F57: ??? (in /sbin/rsyslogd)

==2375==    by 0x1270A0: msgAddJSON (in /sbin/rsyslogd)
==2375==    by 0x7242F2D: ??? (in

/lib64/rsyslog/mmnormalize.so)

==2375==    by 0x14EA64: ??? (in /sbin/rsyslogd)

==2375==    by 0x14F561: ??? (in /sbin/rsyslogd)
==2375==    by 0x14F923: ??? (in /sbin/rsyslogd)
==2375==    by 0x146AB6: ??? (in /sbin/rsyslogd)
==2375==    by 0x1473DA: ??? (in /sbin/rsyslogd)
==2375==    by 0x156FE3: ??? (in /sbin/rsyslogd)
==2375==    by 0x1457D8: ??? (in /sbin/rsyslogd)
==2375==    by 0x13FB5D: wtiWorker (in /sbin/rsyslogd)
==2375==  Address 0x60000001e is not stack'd, malloc'd or

(recently)

free'd

==2375==
==2375==
==2375== Process terminating with default action of signal

(SIGSEGV)

==2375==  Access not within mapped region at address

0x60000001E

==2375==    at 0x5660778: fjson_object_get

(json_object.c:176)

==2375==    by 0x126F57: ??? (in /sbin/rsyslogd)

==2375==    by 0x1270A0: msgAddJSON (in /sbin/rsyslogd)
==2375==    by 0x7242F2D: ??? (in

/lib64/rsyslog/mmnormalize.so)

==2375==    by 0x14EA64: ??? (in /sbin/rsyslogd)

==2375==    by 0x14F561: ??? (in /sbin/rsyslogd)
==2375==    by 0x14F923: ??? (in /sbin/rsyslogd)
==2375==    by 0x146AB6: ??? (in /sbin/rsyslogd)
==2375==    by 0x1473DA: ??? (in /sbin/rsyslogd)
==2375==    by 0x156FE3: ??? (in /sbin/rsyslogd)
==2375==    by 0x1457D8: ??? (in /sbin/rsyslogd)
==2375==    by 0x13FB5D: wtiWorker (in /sbin/rsyslogd)
==2375==  If you believe this happened as a result of a

stack

==2375==  overflow in your program's main thread (unlikely

but

==2375==  possible), you can try to increase the size of the

==2375==  main thread stack using the --main-stacksize=

flag.

==2375==  The main thread stack size used in this run was

10485760.

==2375==

==2375== HEAP SUMMARY:
==2375==     in use at exit: 2,611,014 bytes in 11,732

blocks

==2375==   total heap usage: 27,013 allocs, 15,281 frees,

8,524,299

bytes

allocated

==2375==
==2375== LEAK SUMMARY:
==2375==    definitely lost: 4,117 bytes in 11 blocks
==2375==    indirectly lost: 24,922 bytes in 20 blocks
==2375==      possibly lost: 4,464 bytes in 12 blocks
==2375==    still reachable: 2,577,511 bytes in 11,689

blocks

==2375==         suppressed: 0 bytes in 0 blocks

==2375== Rerun with --leak-check=full to see details of

leaked

memory

==2375==

==2375== For counts of detected and suppressed errors, rerun

with:

-v

==2375== Use --track-origins=yes to see where uninitialised

values

come

from

==2375== ERROR SUMMARY: 5 errors from 3 contexts

(suppressed:

from 7)

[root valgrind-3.11.0]# rsyslogd -v
rsyslogd 8.20.0, compiled with:
PLATFORM: x86_64-redhat-linux-gnu
PLATFORM (lsb_release -d):
FEATURE_REGEXP: Yes
GSSAPI Kerberos 5 support: No
FEATURE_DEBUG (debug build, slow code): No
32bit Atomic operations supported: Yes
64bit Atomic operations supported: Yes
memory allocator: system default
Runtime Instrumentation (slow code): No
uuid support: Yes
Number of Bits in RainerScript integers: 64

See http://www.rsyslog.com for more information.

[root valgrind-3.11.0]# dmesg
..
rs:main Q:Reg[1776]: segfault at 65 ip 00007f0769bcc2b6 sp

00007f0762de87e8

error 4 in libc-2.12.so[7f0769aa4000+18a000]

Thanks,

Alec

On Wed, Jul 20, 2016 at 11:32 PM, Rainer Gerhards <

[email protected]>

wrote:

2016-07-20 23:02 GMT+02:00 Alec Swan <[email protected]>:

I used test RPMs to upgrade from 8.19.0 to 8.20.0.

However,

now

when I

start rsyslog it logs the following error message to

dmesg

and

shuts

rsyslog down:

"rs:main Q:Reg[23801]: segfault at 1a ip 00007fa95ac9f778

sp

00007fa94f5fd878 error 6 in

libfastjson.so.4.0.0[7fa95ac9c000+9000]"

These are the packages that were installed/updated:
Jul 20 20:53:11 yum[22693]: Installed:

libfastjson4-0.99.3-2.el6.x86_64

Jul 20 20:53:11 yum[22693]: Updated:

rsyslog-8.20.0-1.el6.x86_64

Jul 20 20:53:11 yum[22693]: Updated:

rsyslog-mmutf8fix-8.20.0-1.el6.x86_64

Jul 20 20:53:11 yum[22693]: Updated:

rsyslog-mmnormalize-8.20.0-1.el6.x86_64

Jul 20 20:53:11 yum[22693]: Updated:
rsyslog-elasticsearch-8.20.0-1.el6.x86_64


would it be possible that you do a run under valgrind

control?

Rainer



Thanks,

Alec

On Wed, Jul 20, 2016 at 12:11 PM, Michael Biebl <

[email protected]>

wrote:


2016-07-12 23:12 GMT+02:00 David Lang <[email protected]>:

On Tue, 12 Jul 2016, Michael Biebl wrote:

Fwiw, this is the first release I felt comfortable

enough

uploading

to

Debian unstable.

It just hit the archive a few hours ago:
https://tracker.debian.org/pkg/libfastjson

I hope we don't break the API that much anymore

before

1.0.0.

I'm now waiting for a liblognorm 1.1.4 release which

compiles

against

libfastjson 0.99.3.



My understanding is that we are going to get

liblognorm 2

this

cycle

(after

being delayed for the last couple of cycles)


Ok, so what's the plan here? Which liblognorm version

should I

pick?

At least they currently released version doesn't compile

against

the

latest libfastjson release (which is not a surprise,

after

the

API

rework, which I'm thankful for).


Regards,
Michael

--
Why is it that all of the instruments seeking

intelligent

life

in

the

universe are pointed away from Earth?

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow

https://twitter.com/rgerhards

NOTE WELL: This is a PUBLIC mailing list, posts are

ARCHIVED

by a

myriad

of sites beyond our control. PLEASE UNSUBSCRIBE and DO

NOT

POST if

you

DON'T LIKE THAT.


_______________________________________________

rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow

https://twitter.com/rgerhards

NOTE WELL: This is a PUBLIC mailing list, posts are

ARCHIVED

by

myriad

of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT

POST if

you

DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow

https://twitter.com/rgerhards

NOTE WELL: This is a PUBLIC mailing list, posts are

ARCHIVED

by a

myriad

of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT

POST if

you

DON'T LIKE THAT.


_______________________________________________

rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow

https://twitter.com/rgerhards

...

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] libfastjson 0.99.3 released

Reply via email to