Re: [rsyslog] rsyslogd 8.29.0 - crash when omrelp server port is not open + build error in imptcp

2017-08-22 Thread Thomas Deutschmann via rsyslog 
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

On 2017-08-21 13:21, Andreas Wehrmann via rsyslog wrote:
>> actionProcessMessage() always returns RS_RET_ACTION_FAILED (which is
>> -2123),
>> meaning i will be decremented for re-submit and the loop starts all
>> over again.

But this is not the problem itself. It *should* retry. See
action.resumeRetryCount and action.resumeInterval
documentation [1].


> So I ultimately traced it down to this change:
> https://github.com/rsyslog/rsyslog/commit/128214fffac7dcec69b5c8dffdb8222bbd29af27
> 
> 
> Reverting this makes everything seem to work as expected,
> though it probably introduces the bug it was supposed to fix...

This should only trigger the problem, not causing the problem.


My current theory: _No_ msg can be delivered. Even internal messages
nowadays [2]. This will eat up your memory until rsyslogd segfaults:

For example, when rsyslogd launches it will already create a message
like

> rsyslogd:  [origin software="rsyslogd" swVersion="8.29.0" x-pid="31213" 
> x-info="http://www.rsyslog.com;] start

Now this message has to go through your queue. Once it reaches
your omrelp action which is failing, rsyslogd will generate a

> rsyslogd: action 'action 1' suspended, next retry is 

message. This message will also go through your queue and be
processed by the still failing omrelp action (which will be skipped
like you can see in your logs: next retry (if applicable):
966135869 [now 966135839]).

Due to the small amount of memory you just need a small bunch of
message to run out of memory.

And don't forget about the main queue [3].


Do you get some core dumps when rsyslogd segfaults? Maybe you have to
enable this first on your system.

Or could you rsyslogd through gdb/valgrind?

What's about your dmesg? Any messages indicating OOM killer activity?


See also:
=
[1] http://www.rsyslog.com/doc/v8-stable/configuration/actions.html

[2] http://www.rsyslog.com/rsyslog-error-reporting-improved/

[3] http://www.rsyslog.com/doc/v8-stable/concepts/queues.html


- -- 
Regards,
Thomas

-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0

iQJ8BAEBCgBmBQJZnLq0XxSAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w
ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXQzM0M1ODQ4MkM0MDIyOTJEMkUzQzVDMDY5
NzA5RjkwQzNDOTZGRkM4AAoJEJcJ+Qw8lv/IGCQP/3rqvkkXxT2RL++kKawicJWc
DanVorfO+wl94CQScTQZIf+p+DPb0H7QaPh6GvzHN5VO13546+eQgQRn7ndO0ZBR
29qXBf3l2jozaa0xyg2FDmGKrm8d1chzrYutbpWDQCpTlCdXWJYfZOT7SYvGj6LO
Kvuj/pifz3r6DoV6luZIBAde9IcCyGe/JSvzbyEUHWB2jcVdRQShOGt7mFUYlBB6
YFW+CxaEXsC9Kzu7gHWxf6XtB1duNP0l9m1zL2xu4KtU8R1DVxKQeIvIk34JCNsS
9ng5A2e52/5vBeAHw4lgCXUbuNOxtJHtGEwqyE3Re0dgHqd347CqtKY7vx/mM76I
+aXJjzPt4/qgj0t0mrLb/7YVr5tNSoK91aZaSvPLyb4nHMwAjUsjuYMjvfjgXxic
6GQ5m6y+bGLKDDXLi14DVMO7zO2Jv2WQNEvv7NQVTSg0LMv1NIUpCNmORIlLbpyJ
H6LQJtv70e0jNNOOdWrAI6hNkArEKkUiT5fEpUGUUWywY9spKHr3m2iNC4zs8cNp
Bm0uDTV9tVybb58+6paVUsAM+sMrxPBQ+rWvi1C2eLmb6VUaC97OPl0LC0MLPJh5
Eo5V6fI2llsVyjBc+tO/1H/HgusBaynNcwxVy1df9O1Az1ELFOyZFiHxYaO8EEG0
Y/Hx7AtzsBD0yDFjhZjc
=hon9
-END PGP SIGNATURE-
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] feedback request: debug log improvements

2017-08-22 Thread Jan Gerhards 
Many thanks for the suggestions!
I would like to point out that enabling selective debug is already
possible since version 8.29 by using the parameters debug.file and
debug.whitelist. You can find the documentation regarding those
parameters at

http://www.rsyslog.com/doc/v8-stable/rainerscript/global.html

I plan on summarizing the feedback in a short article in the closer
future, but would still be glad about additional feedback.

Jan

2017-08-22 10:20 GMT+02:00 mostolog--- via rsyslog :
> My 2 cents:
>
> As Rsyslog is used to handle several log formats, and encourage to use them,
> it would be great it also use a well-known format.
> Adding timestamps, caller (which component produces the log), thread-id or
> an identifier to "trace" an event going from input to output (print message
> id) would be great.
>
> Perhaps I would test a depth value instead of a loglevel. Enabling selective
> debug (specific modules only) would also be great.
>
> Regards.
>
>
>
> On 14/08/17 10:59, Jan Gerhards wrote:
>>
>> 2017-08-12 23:39 GMT+02:00 Joe Blow via rsyslog
>> :
>>>
>>> Is the output already in JSON? If not it'd be great to have something
>>> that
>>> you could index and query directly.
>>
>> Right now the output isn't in JSON. It is only accessible in the
>> debug-log and only if debugging is enabled. However, I'm unsure about
>> the benefits of having it in JSON format, since the messages
>> themselves have no structure. The only index-able parts would be the
>> (relative) timestamp, the thread and the file in which the output is
>> generated.
>>
>> Because of this, I don't see many uses for debug-output in JSON,
>> especially taking into consideration that the most important parts are
>> the message itself. However, I would also be interested in hearing
>> your opinion about the possible cases where JSON could be used.
>>
>> Jan
>>
>>> Cheers,
>>>
>>> JB
>>>
>>> On Fri, Aug 11, 2017 at 9:13 AM, Jan Gerhards 
>>> wrote:
>>>
 Hi all,

 I am working on a refactoring of the debug output generated by rsyslog
 to increase usability not only for developers, but also for end users.

 For more information see
  https://www.linkedin.com/pulse/improving-rsyslog-debug-
 output-jan-gerhards

 As of rsyslog version 8.29 I have started by implementing first
 configuration options. As mentioned in the article, I would also be
 glad to hear from you about your ideas.

 Jan
>>
>> ___
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
>> LIKE THAT.
>
>
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] feedback request: debug log improvements

2017-08-22 Thread mostolog--- via rsyslog 

My 2 cents:

As Rsyslog is used to handle several log formats, and encourage to use 
them, it would be great it also use a well-known format.
Adding timestamps, caller (which component produces the log), thread-id 
or an identifier to "trace" an event going from input to output (print 
message id) would be great.


Perhaps I would test a depth value instead of a loglevel. Enabling 
selective debug (specific modules only) would also be great.


Regards.


On 14/08/17 10:59, Jan Gerhards wrote:

2017-08-12 23:39 GMT+02:00 Joe Blow via rsyslog :

Is the output already in JSON? If not it'd be great to have something that
you could index and query directly.

Right now the output isn't in JSON. It is only accessible in the
debug-log and only if debugging is enabled. However, I'm unsure about
the benefits of having it in JSON format, since the messages
themselves have no structure. The only index-able parts would be the
(relative) timestamp, the thread and the file in which the output is
generated.

Because of this, I don't see many uses for debug-output in JSON,
especially taking into consideration that the most important parts are
the message itself. However, I would also be interested in hearing
your opinion about the possible cases where JSON could be used.

Jan


Cheers,

JB

On Fri, Aug 11, 2017 at 9:13 AM, Jan Gerhards  wrote:


Hi all,

I am working on a refactoring of the debug output generated by rsyslog
to increase usability not only for developers, but also for end users.

For more information see
 https://www.linkedin.com/pulse/improving-rsyslog-debug-
output-jan-gerhards

As of rsyslog version 8.29 I have started by implementing first
configuration options. As mentioned in the article, I would also be
glad to hear from you about your ideas.

Jan

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.