Re: [rsyslog] Problems with rsyslog Versions > 8.16 and Thread-Handling

[David Lang]

is the version you are using linked to json-c or libfastjson?

we know there were thread-safe problems as a result of json-c

David Lang

On Thu, 12 Jan 2017, Christopher Racky via rsyslog wrote:


Date: Thu, 12 Jan 2017 12:43:31 +0100
From: Christopher Racky via rsyslog 
To: rsyslog-users 
Cc: Christopher Racky 
Subject: [rsyslog] Problems with rsyslog Versions > 8.16 and Thread-Handling

Hello,

I still have big issues with different Servers running RHEL 6.8 (incl.
latest updates) and rsyslog > 8.16.

While 8.16 works fine, all following versions, including 8.24 which
was released a few days ago leads to problems that results in
core-dumps:
rsyslogd[1408]: segfault at 7f9f2910b9d0 ip 7f9f2e18e213 sp
7ffd3d9fe080 error 4 in libpthread-2.12.so[7f9f2e186000+17000]
rsyslogd[4201]: segfault at 7f991c0789d0 ip 7f99210fb213 sp
7ffec5155240 error 4 in libpthread-2.12.so[7f99210f3000+17000]
rsyslogd[2783]: segfault at 7f5f695399d0 ip 7f5f6e5bd213 sp
7ffe4b4da5c0 error 4 in libpthread-2.12.so[7f5f6e5b5000+17000]
rsyslogd[6816]: segfault at 7f519ad269d0 ip 7f519fdaa213 sp
7ffd22bf2470 error 4 in libpthread-2.12.so[7f519fda2000+17000]


My hope was that this topic was solved by
https://github.com/rsyslog/rsyslog/pull/1274

But it was not.
As problems seems not (directly) related to lookup-table, I guess it
also has a problem with thread handling.
https://github.com/rsyslog/rsyslog/issues/1071

Do you have any further idea?

With single-thread mode problem does not appear and also in lab, I was
not able to reproduce it.
But on several production servers the issue occures quite often.


regards
Chris
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] Problems with rsyslog Versions > 8.16 and Thread-Handling

[Christopher Racky via rsyslog]

Hello,

I still have big issues with different Servers running RHEL 6.8 (incl.
latest updates) and rsyslog > 8.16.

While 8.16 works fine, all following versions, including 8.24 which
was released a few days ago leads to problems that results in
core-dumps:
rsyslogd[1408]: segfault at 7f9f2910b9d0 ip 7f9f2e18e213 sp
7ffd3d9fe080 error 4 in libpthread-2.12.so[7f9f2e186000+17000]
rsyslogd[4201]: segfault at 7f991c0789d0 ip 7f99210fb213 sp
7ffec5155240 error 4 in libpthread-2.12.so[7f99210f3000+17000]
rsyslogd[2783]: segfault at 7f5f695399d0 ip 7f5f6e5bd213 sp
7ffe4b4da5c0 error 4 in libpthread-2.12.so[7f5f6e5b5000+17000]
rsyslogd[6816]: segfault at 7f519ad269d0 ip 7f519fdaa213 sp
7ffd22bf2470 error 4 in libpthread-2.12.so[7f519fda2000+17000]


My hope was that this topic was solved by
https://github.com/rsyslog/rsyslog/pull/1274

But it was not.
As problems seems not (directly) related to lookup-table, I guess it
also has a problem with thread handling.
https://github.com/rsyslog/rsyslog/issues/1071

Do you have any further idea?

With single-thread mode problem does not appear and also in lab, I was
not able to reproduce it.
But on several production servers the issue occures quite often.


regards
Chris
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Out of memory

[David Lang]

what rainer says

2G of memory isn't _that_ much nowdays. If you have a 1K log message that then 
gets parsed into a bunch of variables, it's going to take substantially more ram 
(2-3K would not be unreasonable, more is possible)


As the messages are then sent out, they remain in the queue while a copy is made 
and formatted for delivery. That copy can then get compressed/encrypted which 
makes an additional copy.


If each message takes 4K of ram, and you have 100K messages in a queue, you now 
have used 400M of ram, a pretty significant fraction of your 2G total (and this 
doesn't count for code space.


If your queues back up, you can end up eating a LOT of ram, even if you don't 
set omfile buffers large


each outbound connection you have will have it's own buffer, each inbound 
connection will have it's own buffer (amoung other things, this is part of why 
we were trying to discourage you from having separate ports for each log type 
:-)


but the impstats output will show you how large the queues are and what the 
overall resource usage of rsyslog is.


you may also want to look at the output of top (with the 'H' option to see 
threads) and see if you have an explosion of threads happening. I tripped over 
this with dynafiles and async writing when I was trying to improve compressed 
file output performance.


David Lang


On Thu, 12 Jan 2017, Rainer Gerhards wrote:


Have a look what impstats tells you. I guess some queue is defined very
large and fills up.

Rainer

2017-01-12 12:01 GMT+01:00 mostolog--- via rsyslog <
rsyslog@lists.adiscon.com>:


Hi


At this moment, our testing rsyslog is receiving messages from relay
through RELP in RFC3164 format with message in JSON like:

  DATE RELAYHOST TAG: {"field":"foo",..."data":"orig
inal_forwarded_message"}

Once received, mmjsonparse is executed over MSG to check if incoming
messages "comply", before mmnormalize on data (actually doing nothing due
to liblognorm bugs already reported) and setting some other fields.

After that, omelasticsearch action is executed and messages are indexed.

We are receiving ~1 million events/hour and rsyslog has 2GB of memory and
process is killed cause out of memory (every other hour).

What steps could we take to diagnose if that's correct, how to avoid
it...? Any up to date literature?


Thanks. Regards

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Out of memory

[Rainer Gerhards]

Have a look what impstats tells you. I guess some queue is defined very
large and fills up.

Rainer

2017-01-12 12:01 GMT+01:00 mostolog--- via rsyslog <
rsyslog@lists.adiscon.com>:

> Hi
>
>
> At this moment, our testing rsyslog is receiving messages from relay
> through RELP in RFC3164 format with message in JSON like:
>
>   DATE RELAYHOST TAG: {"field":"foo",..."data":"orig
> inal_forwarded_message"}
>
> Once received, mmjsonparse is executed over MSG to check if incoming
> messages "comply", before mmnormalize on data (actually doing nothing due
> to liblognorm bugs already reported) and setting some other fields.
>
> After that, omelasticsearch action is executed and messages are indexed.
>
> We are receiving ~1 million events/hour and rsyslog has 2GB of memory and
> process is killed cause out of memory (every other hour).
>
> What steps could we take to diagnose if that's correct, how to avoid
> it...? Any up to date literature?
>
>
> Thanks. Regards
>
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] Out of memory

[mostolog--- via rsyslog]

Hi


At this moment, our testing rsyslog is receiving messages from relay 
through RELP in RFC3164 format with message in JSON like:


  DATE RELAYHOST TAG: 
{"field":"foo",..."data":"original_forwarded_message"}


Once received, mmjsonparse is executed over MSG to check if incoming 
messages "comply", before mmnormalize on data (actually doing nothing 
due to liblognorm bugs already reported) and setting some other fields.


After that, omelasticsearch action is executed and messages are indexed.

We are receiving ~1 million events/hour and rsyslog has 2GB of memory 
and process is killed cause out of memory (every other hour).


What steps could we take to diagnose if that's correct, how to avoid 
it...? Any up to date literature?



Thanks. Regards

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.