[rsyslog] Merge to Master?

2017-01-03 Thread Brian Knox via rsyslog 
I have a PR for omczmq and for omelasticsearch for the next release - I
just wanted to double check that we merge to master now in the absence of
master-candidate.

Happy New Year!
Brian
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] Omkafka error log json addition

2017-01-03 Thread Aaron Ilovici 
Hello all,

I am working on a patch that will add intended topic name to the errorFile 
produced by the omkafka module in the writeDataError function called from the 
deliveryCallback.

The question I have is how to get the dynamic topic name inside either the 
writeDataError or deliveryCallback functions.

The non-dynamic topic name is in the instance data so I can do:
jval = json_object_new_string(rd_kafka_topic_name(pData->pTopic));
json_object_object_add(json, "topic", jval);

Is there an easy way to access the post-template / intended topic name so I can 
just put this in an if/else for the output JSON?

Thank you for any guidance,



AARON ILOVICI
Software Engineer
Marketing Engineering

[escription: wayfair.com]

WAYFAIR
4 Copley Place
Boston, MA 02116
(617) 532-6100 x1231
ailov...@wayfair.com


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] segfault in libfastjson.so.4.0.0 (with workaround)

2017-01-03 Thread mostolog--- via rsyslog 

Could you create an issue for this on github?

could you also paste some related lines from /opt/secrets/log/8.8.8.8.log ?


El 03/01/17 a las 13:54, Lennard Klein escribió:

Hi list,

I've ran into a segfault in libfastjson.

First some details:
RHEL7 using adiscon repositories:
libfastjson4.x86_64 0.99.4-1.el7
rsyslog.x86_64 8.23.0-1.el7

The following bit of config segfaults rsyslog, as soon as it starts processing 
the first message (its own startup message):
if( $!rfc5424-sd!origin!ip == '8.8.8.8' ) then {
action(type="omfile" file="/opt/secrets/log/8.8.8.8.log" 
template="RSYSLOG_DebugFormat")
& stop
}

Running rsyslog -dn, this is the snippet just before the segfault:
4184.634573199:main Q:Reg/w0  : IF
4184.634578377:main Q:Reg/w0  : var '!rfc5424-sd!origin!ip'
4184.634585979:main Q:Reg/w0  :   ==
4184.634591759:main Q:Reg/w0  : string '172.22.222.17'
4184.634601617:main Q:Reg/w0  : eval expr 0x7f615a4b2020, type 'CMP_EQ'
4184.634604257:main Q:Reg/w0  : eval expr 0x7f615a49ec90, type 'V'

Comparing this to some other debugging output I'm guessing the next step in 
processing is resolving the variable.
As I had a theory the segfault might be due to nothing being in the JSON tree 
yet, I Added the following line just before the if-statement:
set $!foo = "bar";

Which resulted in the message being processed as expected.

As I'm not at all familiar with the rsyslog code base, this is as far as I can 
get in my analysis at the moment. Does anyone have suggestions on how to 
proceed?

Regards,
Lennard Klein
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] segfault in libfastjson.so.4.0.0 (with workaround)

2017-01-03 Thread Lennard Klein 
Hi list,

I've ran into a segfault in libfastjson.

First some details:
RHEL7 using adiscon repositories:
libfastjson4.x86_64 0.99.4-1.el7
rsyslog.x86_64 8.23.0-1.el7

The following bit of config segfaults rsyslog, as soon as it starts processing 
the first message (its own startup message):
if( $!rfc5424-sd!origin!ip == '8.8.8.8' ) then {
action(type="omfile" file="/opt/secrets/log/8.8.8.8.log" 
template="RSYSLOG_DebugFormat")
& stop
}

Running rsyslog -dn, this is the snippet just before the segfault:
4184.634573199:main Q:Reg/w0  : IF
4184.634578377:main Q:Reg/w0  : var '!rfc5424-sd!origin!ip'
4184.634585979:main Q:Reg/w0  :   ==
4184.634591759:main Q:Reg/w0  : string '172.22.222.17'
4184.634601617:main Q:Reg/w0  : eval expr 0x7f615a4b2020, type 'CMP_EQ'
4184.634604257:main Q:Reg/w0  : eval expr 0x7f615a49ec90, type 'V'

Comparing this to some other debugging output I'm guessing the next step in 
processing is resolving the variable.
As I had a theory the segfault might be due to nothing being in the JSON tree 
yet, I Added the following line just before the if-statement:
set $!foo = "bar";

Which resulted in the message being processed as expected.

As I'm not at all familiar with the rsyslog code base, this is as far as I can 
get in my analysis at the moment. Does anyone have suggestions on how to 
proceed?

Regards,
Lennard Klein
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Handling "log explosions"

2017-01-03 Thread mostolog--- via rsyslog 


If you know that a log message is uninteresting, then you want to 
throw it away, but count how many times it happened because the number 
of times that an uninteresting log happens can be interesting.

That's exactly what I'm looking for




1.rsyslog gets message
2.if it's a notifiable error
 if it's already "notified", increase notify counter
what do you do with the notify counter? do you want it spit out along 
with other stats (in which case dyn_stats is the right answer) or do 
you want to do something else?

I just though to use it as drop criteria. I was thinking something like:

#specific error msg received
increase counter
If counter>=5
  drop message (it will be reported later)
else
  forward

#specific ok message received
counter=0
forward

#scheduled task (each X minutes)
report current_counter
counter=0




   drop message (don't log it)
 else #not "notified"
   notify error
   log message
is there anything that resets "not notified"? or do you only want one 
notification per syslog startup.

See above




3.if it's a recovery
   log recovery
   notify recovered
4.otherwise, log normally

In the above, notification could be a snmptrap to our monitoring 
system, and "if notified" could be a "global" errorCount variable or 
something similar...
keep in mind there are the global variables $\ that you can use for 
this sort of flag, but checking them is relatively expensive, so you 
should think about what you are really wanting here.


It may be good enough to not do any tracking of 'already notified' and 
instead just do


if  then increase counter

and then spit the counters out to your monitoring system. If there 
were no messages of that type, you have no message to your monitoring 
system. If there were messages of that type, you have a notification 
of how many times it happened that monitoring period. If the 
monitoring period is relatively short (say 1-5 min), this may be 
sufficient for your system.
Consider that I'm also trying to save "disk space" ie: not storing huge 
amount of "droppeable" events


Seems dyn_stats is a good alternative, but I haven't time yet to play 
with it.

Regards
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.