Re: [rsyslog] Load balancing issue
First of all i'm ashamed to tell you what's coming :p
Don't blame me.
What do you think of the following workaround ?
My relp log flow :
ss -lap -o state established \( dport = :20514 \)
Fri Jul 24 13:50:25 2015
Recv-Q Send-Q Local Address:Port Peer Address:Port
0 0 10.17.252.4:4642710.19.12.8:20514
users:((haproxy,15625,2))
0 0127.0.0.1:59709 127.0.0.1:20514
users:((rsyslogd,26947,2))
0 22816 10.17.252.4:4495310.19.12.9:20514
timer:(on,012ms,0) users:((haproxy,15625,10))
1512 27160127.0.0.1:59711 127.0.0.1:20514
timer:(on,204ms,0) users:((rsyslogd,26947,10))
#/bin/bash
established=$(ss -lnp -o state established \( dport = :20514 \) not dst
127.0.0.1 | tail -n +2 | awk '{print $3}')
for socket in ${established};do
#echo estab = ${socket}
IFS=':' read -a socket_arr ${socket}
#echo ip = ${socket_arr[0]}, port = ${socket_arr[1]}
tcpkill -i eth0 host ${socket_arr[0]} and port ${socket_arr[1]}
tcpkillpid=( ${tcpkillpid} $! )
done
sleep 2
kill ${tcpkillpid[@]}
If the kill -HUP doesn't work, as it is relp protocole, i think i still have a
reliable log forwarding, right ?
Regards,
Smana
- Mail original -
De: smain...@free.fr
À: rsyslog-users rsyslog@lists.adiscon.com
Envoyé: Vendredi 24 Juillet 2015 13:48:53
Objet: Re: [rsyslog] Load balancing issue
yes that was exactly what i was running :
while true; do
kill -HUP $(pgrep rsyslogd); sleep 5;
done
watch 'ss -lap -o state established \( dport = :20514 \)'
Recv-Q Send-Q Local Address:Port Peer Address:Port
0 0 10.17.252.4:6044510.19.12.5:20514
users:((haproxy,15625,10))
0 6504 10.17.252.4:5912610.19.12.9:20514
timer:(on,012ms,0) users:((haproxy,15625,2))
0 48816127.0.0.1:45659 127.0.0.1:20514
timer:(persist,192ms,0) users:((rsyslogd,26947,2))
1760 5424 127.0.0.1:45651 127.0.0.1:20514
timer:(on,204ms,0) users:((rsyslogd,26947,5))
After a few seconds (say 30), the destination hosts stay the same. After HUP.
Recv-Q Send-Q Local Address:Port Peer Address:Port
0 1432 10.17.252.4:6044510.19.12.5:20514
timer:(on,176ms,0) users:((haproxy,15625,10))
0 16296 10.17.252.4:5912610.19.12.9:20514
timer:(on,012ms,0) users:((haproxy,15625,2))
0 48816127.0.0.1:45659 127.0.0.1:20514
timer:(persist,172ms,0) users:((rsyslogd,26947,2))
67221728127.0.0.1:45651 127.0.0.1:20514
timer:(on,204ms,0) users:((rsyslogd,26947,5))
Regards,
Smana
- Mail original -
De: David Lang da...@lang.hm
À: rsyslog-users rsyslog@lists.adiscon.com
Envoyé: Vendredi 24 Juillet 2015 13:38:37
Objet: Re: [rsyslog] Load balancing issue
On Fri, 24 Jul 2015, smain...@free.fr wrote:
Hi David,
Thank you again.
I tried the workaround but the HUP signal doesn't close the outputs :
* the source pid, source port, dest port stay the same even with a kill -HUP
loop.
In my logs i can see that the configuration is reread but the output's aren't
closed.
Jul 24 09:52:38 log-aggregator-itx2-1 rsyslogd: [origin software=rsyslogd
swVersion=8.10.0 x-pid=23317 x-info=http://www.rsyslog.com;] rsyslogd
was HUPed
Jul 24 09:52:43 log-aggregator-itx2-1 rsyslogd: [origin software=rsyslogd
swVersion=8.10.0 x-pid=23317 x-info=http://www.rsyslog.com;] rsyslogd
was HUPed
Jul 24 09:52:48 log-aggregator-itx2-1 rsyslogd: [origin software=rsyslogd
swVersion=8.10.0 x-pid=23317 x-info=http://www.rsyslog.com;] rsyslogd
was HUPed
none of this data is supposed to chenage, where are you looking to say that the
source port is not changing?
do a netstat -an |grep ESTAB to show the established connections before and
after the HUP
David Lang
Regards,
Smana
- Mail original -
De: David Lang da...@lang.hm
À: rsyslog-users rsyslog@lists.adiscon.com
Envoyé: Jeudi 23 Juillet 2015 20:18:26
Objet: Re: [rsyslog] Load balancing issue
On Thu, 23 Jul 2015, smain...@free.fr wrote:
So to resume there's 2 remaining issues i would like to address :
* disabling octet count causes that my condition based on appname to be
ignored
see my other message about fixing the format.
* i have a segfault when i enable rebindInterval
this looks like a bug that's been reported and someone has been assigned to
work onit. I'd suggest watching this bug and posting if alorbach has any
trouble
duplicating things
https://github.com/rsyslog/rsyslog/issues/120
In the meantime, as a poor-man's work-around, you can setup a script that
does
Re: [rsyslog] Load balancing issue
On Fri, 24 Jul 2015, smain...@free.fr wrote: Hi David, Thank you again. I tried the workaround but the HUP signal doesn't close the outputs : * the source pid, source port, dest port stay the same even with a kill -HUP loop. In my logs i can see that the configuration is reread but the output's aren't closed. Jul 24 09:52:38 log-aggregator-itx2-1 rsyslogd: [origin software=rsyslogd swVersion=8.10.0 x-pid=23317 x-info=http://www.rsyslog.com;] rsyslogd was HUPed Jul 24 09:52:43 log-aggregator-itx2-1 rsyslogd: [origin software=rsyslogd swVersion=8.10.0 x-pid=23317 x-info=http://www.rsyslog.com;] rsyslogd was HUPed Jul 24 09:52:48 log-aggregator-itx2-1 rsyslogd: [origin software=rsyslogd swVersion=8.10.0 x-pid=23317 x-info=http://www.rsyslog.com;] rsyslogd was HUPed none of this data is supposed to chenage, where are you looking to say that the source port is not changing? do a netstat -an |grep ESTAB to show the established connections before and after the HUP David Lang Regards, Smana - Mail original - De: David Lang da...@lang.hm À: rsyslog-users rsyslog@lists.adiscon.com Envoyé: Jeudi 23 Juillet 2015 20:18:26 Objet: Re: [rsyslog] Load balancing issue On Thu, 23 Jul 2015, smain...@free.fr wrote: So to resume there's 2 remaining issues i would like to address : * disabling octet count causes that my condition based on appname to be ignored see my other message about fixing the format. * i have a segfault when i enable rebindInterval this looks like a bug that's been reported and someone has been assigned to work onit. I'd suggest watching this bug and posting if alorbach has any trouble duplicating things https://github.com/rsyslog/rsyslog/issues/120 In the meantime, as a poor-man's work-around, you can setup a script that does something like while true do killall -HUP rsyslogd sleep 10 done this will send the HUP signal to rsyslog every 10 seconds, which will cause it to close it's outputs (the same thing the rebindinterval does every X messages) David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Load balancing issue
yes that was exactly what i was running : while true; do kill -HUP $(pgrep rsyslogd); sleep 5; done watch 'ss -lap -o state established \( dport = :20514 \)' Recv-Q Send-Q Local Address:Port Peer Address:Port 0 0 10.17.252.4:6044510.19.12.5:20514 users:((haproxy,15625,10)) 0 6504 10.17.252.4:5912610.19.12.9:20514 timer:(on,012ms,0) users:((haproxy,15625,2)) 0 48816127.0.0.1:45659 127.0.0.1:20514 timer:(persist,192ms,0) users:((rsyslogd,26947,2)) 1760 5424 127.0.0.1:45651 127.0.0.1:20514 timer:(on,204ms,0) users:((rsyslogd,26947,5)) After a few seconds (say 30), the destination hosts stay the same. After HUP. Recv-Q Send-Q Local Address:Port Peer Address:Port 0 1432 10.17.252.4:6044510.19.12.5:20514 timer:(on,176ms,0) users:((haproxy,15625,10)) 0 16296 10.17.252.4:5912610.19.12.9:20514 timer:(on,012ms,0) users:((haproxy,15625,2)) 0 48816127.0.0.1:45659 127.0.0.1:20514 timer:(persist,172ms,0) users:((rsyslogd,26947,2)) 67221728127.0.0.1:45651 127.0.0.1:20514 timer:(on,204ms,0) users:((rsyslogd,26947,5)) Regards, Smana - Mail original - De: David Lang da...@lang.hm À: rsyslog-users rsyslog@lists.adiscon.com Envoyé: Vendredi 24 Juillet 2015 13:38:37 Objet: Re: [rsyslog] Load balancing issue On Fri, 24 Jul 2015, smain...@free.fr wrote: Hi David, Thank you again. I tried the workaround but the HUP signal doesn't close the outputs : * the source pid, source port, dest port stay the same even with a kill -HUP loop. In my logs i can see that the configuration is reread but the output's aren't closed. Jul 24 09:52:38 log-aggregator-itx2-1 rsyslogd: [origin software=rsyslogd swVersion=8.10.0 x-pid=23317 x-info=http://www.rsyslog.com;] rsyslogd was HUPed Jul 24 09:52:43 log-aggregator-itx2-1 rsyslogd: [origin software=rsyslogd swVersion=8.10.0 x-pid=23317 x-info=http://www.rsyslog.com;] rsyslogd was HUPed Jul 24 09:52:48 log-aggregator-itx2-1 rsyslogd: [origin software=rsyslogd swVersion=8.10.0 x-pid=23317 x-info=http://www.rsyslog.com;] rsyslogd was HUPed none of this data is supposed to chenage, where are you looking to say that the source port is not changing? do a netstat -an |grep ESTAB to show the established connections before and after the HUP David Lang Regards, Smana - Mail original - De: David Lang da...@lang.hm À: rsyslog-users rsyslog@lists.adiscon.com Envoyé: Jeudi 23 Juillet 2015 20:18:26 Objet: Re: [rsyslog] Load balancing issue On Thu, 23 Jul 2015, smain...@free.fr wrote: So to resume there's 2 remaining issues i would like to address : * disabling octet count causes that my condition based on appname to be ignored see my other message about fixing the format. * i have a segfault when i enable rebindInterval this looks like a bug that's been reported and someone has been assigned to work onit. I'd suggest watching this bug and posting if alorbach has any trouble duplicating things https://github.com/rsyslog/rsyslog/issues/120 In the meantime, as a poor-man's work-around, you can setup a script that does something like while true do killall -HUP rsyslogd sleep 10 done this will send the HUP signal to rsyslog every 10 seconds, which will cause it to close it's outputs (the same thing the rebindinterval does every X messages) David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http
Re: [rsyslog] Load balancing issue
On Thu, 23 Jul 2015, Brian Knox wrote:
From your diagram, it looks like you are trying to load balance RELP. As
far as I know, RELP does not suppot ActionTCPRebindInterval. I believe
this has been discussed on the mailing list:
http://lists.adiscon.net/pipermail/rsyslog/2013-May/032549.html
Unless something has changed, you need to use the omfwd module if you want
to use tcp rebinding. This isn't a bug - this is documented behavior. The
rebind interval parameter is documented as a parameter for omfwd. RELP
uses omrelp, which has no such paramater. See:
http://www.rsyslog.com/doc/v8-stable/configuration/modules/omfwd.html and
http://www.rsyslog.com/doc/v8-stable/configuration/modules/omrelp.html
with the new style config it does in the current git branch. It looks like it
was added in 7.3.15
/* tables for interfacing with the v6 config system */
/* action (instance) parameters */
static struct cnfparamdescr actpdescr[] = {
{ target, eCmdHdlrGetWord, 1 },
{ tls, eCmdHdlrBinary, 0 },
{ tls.compression, eCmdHdlrBinary, 0 },
{ tls.prioritystring, eCmdHdlrString, 0 },
{ tls.cacert, eCmdHdlrString, 0 },
{ tls.mycert, eCmdHdlrString, 0 },
{ tls.myprivkey, eCmdHdlrString, 0 },
{ tls.authmode, eCmdHdlrString, 0 },
{ tls.permittedpeer, eCmdHdlrArray, 0 },
{ port, eCmdHdlrGetWord, 0 },
{ rebindinterval, eCmdHdlrInt, 0 },
{ windowsize, eCmdHdlrInt, 0 },
{ timeout, eCmdHdlrInt, 0 },
{ localclientip, eCmdHdlrGetWord, 0 },
{ template, eCmdHdlrGetWord, 0 }
};
I use ActionTCPRebindInterval with haproxy with plain TCP. It works very
well.
Cheers,
Brian
On Thu, Jul 23, 2015 at 7:03 AM, smain...@free.fr wrote:
With the architecture enclosed.
- Mail original -
De: smain...@free.fr
À: rsyslog-users rsyslog@lists.adiscon.com
Envoyé: Jeudi 23 Juillet 2015 11:59:35
Objet: [rsyslog] Load balancing issue
Hello all,
I'm currently trying to load balance the log traffic accross several
servers.
I thought my configuration with ActionSendTCPRebindInterval option was
working properly, unfortunately my recent benchs show that the log flow is
not well load balanced.
Please find below a part of the architecture :
My problem is located on the log aggregators : the rsyslog send its
traffic to haproxy on localhost using relp protocol.
I monitored the tcp sessions and i can see that haproxy doesn't change the
destination servers.
watch 'ss -lap -o state established \( dport = :20514 \)'
Recv-Q Send-Q Local Address:Port Peer Address:Port
1716 0 127.0.0.1:43652 127.0.0.1:20514 users:((rsyslogd,8409,88))
0 1138 10.17.252.4:58436 10.19.12.5:20514 timer:(on,196ms,0)
users:((haproxy,3922
,2))
1760 0 127.0.0.1:43650 127.0.0.1:20514 users:((rsyslogd,8409,22))
0 0 10.17.252.4:55583 10.19.12.6:20514 users:((haproxy,3922,10))
Please find enclosed my configuration.
NB :
- the source pid (rsyslog) never change as it is expected with
ActionSendTCPRebindInterval
- i mixed legacy and new syntaxe because of the following bug
https://github.com/rsyslog/rsyslog/issues/96
This bug is annoying and i didn't receive any update since about 4 months
Could you please help me ?
OS : debian7
rsyslog version : 8.10
Regards ,
Smana
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
Re: [rsyslog] Load balancing issue
On Thu, 23 Jul 2015, Brian Knox wrote:
Aha! David - to summarize, is the problem then that:
a) the parameter did not exist previously, and
b) was only added for the new style configs?
I think so.
David Lang
Brian
On Thu, Jul 23, 2015 at 7:59 AM, David Lang da...@lang.hm wrote:
On Thu, 23 Jul 2015, Brian Knox wrote:
From your diagram, it looks like you are trying to load balance RELP. As
far as I know, RELP does not suppot ActionTCPRebindInterval. I believe
this has been discussed on the mailing list:
http://lists.adiscon.net/pipermail/rsyslog/2013-May/032549.html
Unless something has changed, you need to use the omfwd module if you want
to use tcp rebinding. This isn't a bug - this is documented behavior.
The
rebind interval parameter is documented as a parameter for omfwd. RELP
uses omrelp, which has no such paramater. See:
http://www.rsyslog.com/doc/v8-stable/configuration/modules/omfwd.html and
http://www.rsyslog.com/doc/v8-stable/configuration/modules/omrelp.html
with the new style config it does in the current git branch. It looks like
it was added in 7.3.15
/* tables for interfacing with the v6 config system */
/* action (instance) parameters */
static struct cnfparamdescr actpdescr[] = {
{ target, eCmdHdlrGetWord, 1 },
{ tls, eCmdHdlrBinary, 0 },
{ tls.compression, eCmdHdlrBinary, 0 },
{ tls.prioritystring, eCmdHdlrString, 0 },
{ tls.cacert, eCmdHdlrString, 0 },
{ tls.mycert, eCmdHdlrString, 0 },
{ tls.myprivkey, eCmdHdlrString, 0 },
{ tls.authmode, eCmdHdlrString, 0 },
{ tls.permittedpeer, eCmdHdlrArray, 0 },
{ port, eCmdHdlrGetWord, 0 },
{ rebindinterval, eCmdHdlrInt, 0 },
{ windowsize, eCmdHdlrInt, 0 },
{ timeout, eCmdHdlrInt, 0 },
{ localclientip, eCmdHdlrGetWord, 0 },
{ template, eCmdHdlrGetWord, 0 }
};
I use ActionTCPRebindInterval with haproxy with plain TCP. It works very
well.
Cheers,
Brian
On Thu, Jul 23, 2015 at 7:03 AM, smain...@free.fr wrote:
With the architecture enclosed.
- Mail original -
De: smain...@free.fr
À: rsyslog-users rsyslog@lists.adiscon.com
Envoyé: Jeudi 23 Juillet 2015 11:59:35
Objet: [rsyslog] Load balancing issue
Hello all,
I'm currently trying to load balance the log traffic accross several
servers.
I thought my configuration with ActionSendTCPRebindInterval option was
working properly, unfortunately my recent benchs show that the log flow
is
not well load balanced.
Please find below a part of the architecture :
My problem is located on the log aggregators : the rsyslog send its
traffic to haproxy on localhost using relp protocol.
I monitored the tcp sessions and i can see that haproxy doesn't change
the
destination servers.
watch 'ss -lap -o state established \( dport = :20514 \)'
Recv-Q Send-Q Local Address:Port Peer Address:Port
1716 0 127.0.0.1:43652 127.0.0.1:20514 users:((rsyslogd,8409,88))
0 1138 10.17.252.4:58436 10.19.12.5:20514 timer:(on,196ms,0)
users:((haproxy,3922
,2))
1760 0 127.0.0.1:43650 127.0.0.1:20514 users:((rsyslogd,8409,22))
0 0 10.17.252.4:55583 10.19.12.6:20514 users:((haproxy,3922,10))
Please find enclosed my configuration.
NB :
- the source pid (rsyslog) never change as it is expected with
ActionSendTCPRebindInterval
- i mixed legacy and new syntaxe because of the following bug
https://github.com/rsyslog/rsyslog/issues/96
This bug is annoying and i didn't receive any update since about 4 months
Could you please help me ?
OS : debian7
rsyslog version : 8.10
Regards ,
Smana
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC
Re: [rsyslog] Load balancing issue
2015-07-23 15:12 GMT+02:00 David Lang da...@lang.hm:
On Thu, 23 Jul 2015, smain...@free.fr wrote:
1- it solved my issue regarding the disk queue not created. i'll update
the github issue.
good.
2- i have a lot of errors like rsyslogd: Framing Error in received TCP
message: invalid octet count -1871509715. [v8.10.0]
this means that you are getting malformed data sent to you. Rsyslog
implements an extension to the syslog protocol where instead of each log
message being a string of text followed by a newline, the sending system can
send a number at the beginning (instead of PRI where PRI is the combined
facility/severity data) and rsyslog will then read that number of bytes as
the message. This allows a message to contain embedded newlines.
What's happening is that you have something sending you digits at the
beginning of the message, rsyslog is trying to interpret this, but it's
garbage data. I don't know if there is a way to disable octet counted mode
on the reciever or not.
There is a parameter to do that, but I don't remember the name out of my head.
Rainer
similarly, a message starting with 'z' is
interpreted as a compressed message.
The 'best' answer is to figure out which system is generating the invalid
messages and fix it there. If you can do so.
David Lang
3- As soon as i enable the rebindInterval option, rsyslog segfaults
[Thu Jul 23 12:46:03 2015] rs:analytics qu[19247]: segfault at 20 ip
7f3a64efa624 sp 7f3a5b1f5bc8 error 4 in
librelp.so.0.1.0[7f3a64eee000+11000]
Please find the startup debug logs here :
https://gist.github.com/Smana/21f1add821b91f1a0bc1
Regards,
Smana
- Mail original -
De: Brian Knox bk...@digitalocean.com
À: rsyslog-users rsyslog@lists.adiscon.com
Envoyé: Jeudi 23 Juillet 2015 14:17:05
Objet: Re: [rsyslog] Load balancing issue
Aha! David - to summarize, is the problem then that:
a) the parameter did not exist previously, and
b) was only added for the new style configs?
Brian
On Thu, Jul 23, 2015 at 7:59 AM, David Lang da...@lang.hm wrote:
On Thu, 23 Jul 2015, Brian Knox wrote:
From your diagram, it looks like you are trying to load balance RELP. As
far as I know, RELP does not suppot ActionTCPRebindInterval. I believe
this has been discussed on the mailing list:
http://lists.adiscon.net/pipermail/rsyslog/2013-May/032549.html
Unless something has changed, you need to use the omfwd module if you
want
to use tcp rebinding. This isn't a bug - this is documented behavior.
The
rebind interval parameter is documented as a parameter for omfwd. RELP
uses omrelp, which has no such paramater. See:
http://www.rsyslog.com/doc/v8-stable/configuration/modules/omfwd.html
and
http://www.rsyslog.com/doc/v8-stable/configuration/modules/omrelp.html
with the new style config it does in the current git branch. It looks
like
it was added in 7.3.15
/* tables for interfacing with the v6 config system */
/* action (instance) parameters */
static struct cnfparamdescr actpdescr[] = {
{ target, eCmdHdlrGetWord, 1 },
{ tls, eCmdHdlrBinary, 0 },
{ tls.compression, eCmdHdlrBinary, 0 },
{ tls.prioritystring, eCmdHdlrString, 0 },
{ tls.cacert, eCmdHdlrString, 0 },
{ tls.mycert, eCmdHdlrString, 0 },
{ tls.myprivkey, eCmdHdlrString, 0 },
{ tls.authmode, eCmdHdlrString, 0 },
{ tls.permittedpeer, eCmdHdlrArray, 0 },
{ port, eCmdHdlrGetWord, 0 },
{ rebindinterval, eCmdHdlrInt, 0 },
{ windowsize, eCmdHdlrInt, 0 },
{ timeout, eCmdHdlrInt, 0 },
{ localclientip, eCmdHdlrGetWord, 0 },
{ template, eCmdHdlrGetWord, 0 }
};
I use ActionTCPRebindInterval with haproxy with plain TCP. It works
very
well.
Cheers,
Brian
On Thu, Jul 23, 2015 at 7:03 AM, smain...@free.fr wrote:
With the architecture enclosed.
- Mail original -
De: smain...@free.fr
À: rsyslog-users rsyslog@lists.adiscon.com
Envoyé: Jeudi 23 Juillet 2015 11:59:35
Objet: [rsyslog] Load balancing issue
Hello all,
I'm currently trying to load balance the log traffic accross several
servers.
I thought my configuration with ActionSendTCPRebindInterval option
was
working properly, unfortunately my recent benchs show that the log flow
is
not well load balanced.
Please find below a part of the architecture :
My problem is located on the log aggregators : the rsyslog send its
traffic to haproxy on localhost using relp protocol.
I monitored the tcp sessions and i can see that haproxy doesn't change
the
destination servers.
watch 'ss -lap -o state established \( dport = :20514 \)'
Recv-Q Send-Q Local Address:Port Peer Address:Port
1716 0 127.0.0.1:43652 127.0.0.1:20514 users:((rsyslogd,8409,88))
0 1138 10.17.252.4:58436 10.19.12.5:20514 timer:(on,196ms,0)
users:((haproxy,3922
,2))
1760 0 127.0.0.1:43650 127.0.0.1:20514 users:((rsyslogd,8409,22))
0 0 10.17.252.4:55583
Re: [rsyslog] Load balancing issue
From your diagram, it looks like you are trying to load balance RELP. As far as I know, RELP does not suppot ActionTCPRebindInterval. I believe this has been discussed on the mailing list: http://lists.adiscon.net/pipermail/rsyslog/2013-May/032549.html Unless something has changed, you need to use the omfwd module if you want to use tcp rebinding. This isn't a bug - this is documented behavior. The rebind interval parameter is documented as a parameter for omfwd. RELP uses omrelp, which has no such paramater. See: http://www.rsyslog.com/doc/v8-stable/configuration/modules/omfwd.html and http://www.rsyslog.com/doc/v8-stable/configuration/modules/omrelp.html I use ActionTCPRebindInterval with haproxy with plain TCP. It works very well. Cheers, Brian On Thu, Jul 23, 2015 at 7:03 AM, smain...@free.fr wrote: With the architecture enclosed. - Mail original - De: smain...@free.fr À: rsyslog-users rsyslog@lists.adiscon.com Envoyé: Jeudi 23 Juillet 2015 11:59:35 Objet: [rsyslog] Load balancing issue Hello all, I'm currently trying to load balance the log traffic accross several servers. I thought my configuration with ActionSendTCPRebindInterval option was working properly, unfortunately my recent benchs show that the log flow is not well load balanced. Please find below a part of the architecture : My problem is located on the log aggregators : the rsyslog send its traffic to haproxy on localhost using relp protocol. I monitored the tcp sessions and i can see that haproxy doesn't change the destination servers. watch 'ss -lap -o state established \( dport = :20514 \)' Recv-Q Send-Q Local Address:Port Peer Address:Port 1716 0 127.0.0.1:43652 127.0.0.1:20514 users:((rsyslogd,8409,88)) 0 1138 10.17.252.4:58436 10.19.12.5:20514 timer:(on,196ms,0) users:((haproxy,3922 ,2)) 1760 0 127.0.0.1:43650 127.0.0.1:20514 users:((rsyslogd,8409,22)) 0 0 10.17.252.4:55583 10.19.12.6:20514 users:((haproxy,3922,10)) Please find enclosed my configuration. NB : - the source pid (rsyslog) never change as it is expected with ActionSendTCPRebindInterval - i mixed legacy and new syntaxe because of the following bug https://github.com/rsyslog/rsyslog/issues/96 This bug is annoying and i didn't receive any update since about 4 months Could you please help me ? OS : debian7 rsyslog version : 8.10 Regards , Smana ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Load balancing issue
Hi Brian, David,
Thank you for your help.
I changed my configuration as follows :
ruleset(name=forward){
if $programname startswith 'CDN.' then {
action( type=omrelp
name=analytics
target=localhost
#rebindinterval=50
port=20514
queue.size=5000
queue.type=LinkedList
queue.spoolDirectory=/var/spool/rsyslog
queue.filename=analytics-spool
queue.lowwatermark=2000
queue.highwatermark=3500
queue.discardmark=5000
queue.maxfilesize=1g
queue.saveonshutdown=on
action.ResumeInterval=10
action.ResumeRetryCount=-1
action.reportSuspension=on
action.reportSuspensionContinuation=on
)
}
}
1- it solved my issue regarding the disk queue not created. i'll update the
github issue.
2- i have a lot of errors like
rsyslogd: Framing Error in received TCP message: invalid octet count
-1871509715. [v8.10.0]
3- As soon as i enable the rebindInterval option, rsyslog segfaults
[Thu Jul 23 12:46:03 2015] rs:analytics qu[19247]: segfault at 20 ip
7f3a64efa624 sp 7f3a5b1f5bc8 error 4 in
librelp.so.0.1.0[7f3a64eee000+11000]
Please find the startup debug logs here :
https://gist.github.com/Smana/21f1add821b91f1a0bc1
Regards,
Smana
- Mail original -
De: Brian Knox bk...@digitalocean.com
À: rsyslog-users rsyslog@lists.adiscon.com
Envoyé: Jeudi 23 Juillet 2015 14:17:05
Objet: Re: [rsyslog] Load balancing issue
Aha! David - to summarize, is the problem then that:
a) the parameter did not exist previously, and
b) was only added for the new style configs?
Brian
On Thu, Jul 23, 2015 at 7:59 AM, David Lang da...@lang.hm wrote:
On Thu, 23 Jul 2015, Brian Knox wrote:
From your diagram, it looks like you are trying to load balance RELP. As
far as I know, RELP does not suppot ActionTCPRebindInterval. I believe
this has been discussed on the mailing list:
http://lists.adiscon.net/pipermail/rsyslog/2013-May/032549.html
Unless something has changed, you need to use the omfwd module if you want
to use tcp rebinding. This isn't a bug - this is documented behavior.
The
rebind interval parameter is documented as a parameter for omfwd. RELP
uses omrelp, which has no such paramater. See:
http://www.rsyslog.com/doc/v8-stable/configuration/modules/omfwd.html and
http://www.rsyslog.com/doc/v8-stable/configuration/modules/omrelp.html
with the new style config it does in the current git branch. It looks like
it was added in 7.3.15
/* tables for interfacing with the v6 config system */
/* action (instance) parameters */
static struct cnfparamdescr actpdescr[] = {
{ target, eCmdHdlrGetWord, 1 },
{ tls, eCmdHdlrBinary, 0 },
{ tls.compression, eCmdHdlrBinary, 0 },
{ tls.prioritystring, eCmdHdlrString, 0 },
{ tls.cacert, eCmdHdlrString, 0 },
{ tls.mycert, eCmdHdlrString, 0 },
{ tls.myprivkey, eCmdHdlrString, 0 },
{ tls.authmode, eCmdHdlrString, 0 },
{ tls.permittedpeer, eCmdHdlrArray, 0 },
{ port, eCmdHdlrGetWord, 0 },
{ rebindinterval, eCmdHdlrInt, 0 },
{ windowsize, eCmdHdlrInt, 0 },
{ timeout, eCmdHdlrInt, 0 },
{ localclientip, eCmdHdlrGetWord, 0 },
{ template, eCmdHdlrGetWord, 0 }
};
I use ActionTCPRebindInterval with haproxy with plain TCP. It works very
well.
Cheers,
Brian
On Thu, Jul 23, 2015 at 7:03 AM, smain...@free.fr wrote:
With the architecture enclosed.
- Mail original -
De: smain...@free.fr
À: rsyslog-users rsyslog@lists.adiscon.com
Envoyé: Jeudi 23 Juillet 2015 11:59:35
Objet: [rsyslog] Load balancing issue
Hello all,
I'm currently trying to load balance the log traffic accross several
servers.
I thought my configuration with ActionSendTCPRebindInterval option was
working properly, unfortunately my recent benchs show that the log flow
is
not well load balanced.
Please find below a part of the architecture :
My problem is located on the log aggregators : the rsyslog send its
traffic to haproxy on localhost using relp protocol.
I monitored the tcp sessions and i can see that haproxy doesn't change
the
destination servers.
watch 'ss -lap -o state established \( dport = :20514 \)'
Recv-Q Send-Q Local Address:Port Peer Address:Port
1716 0 127.0.0.1:43652 127.0.0.1:20514 users:((rsyslogd,8409,88))
0 1138 10.17.252.4:58436 10.19.12.5:20514 timer:(on,196ms,0)
users:((haproxy,3922
,2))
1760 0 127.0.0.1:43650 127.0.0.1:20514 users:((rsyslogd,8409,22))
0 0 10.17.252.4:55583 10.19.12.6:20514 users:((haproxy,3922,10))
Please find enclosed my configuration.
NB :
- the source pid (rsyslog) never change as it is expected with
ActionSendTCPRebindInterval
- i mixed legacy and new syntaxe because of the following bug
https://github.com
Re: [rsyslog] Load balancing issue
2015-07-23 14:59 GMT+02:00 David Lang da...@lang.hm:
On Thu, 23 Jul 2015, Brian Knox wrote:
Aha! David - to summarize, is the problem then that:
a) the parameter did not exist previously, and
b) was only added for the new style configs?
just to add a general note: we never add new legacy statements. If a
new feature is implemented, you will need to change your config in any
way, so you can also use new style config.
Rainer
I think so.
David Lang
Brian
On Thu, Jul 23, 2015 at 7:59 AM, David Lang da...@lang.hm wrote:
On Thu, 23 Jul 2015, Brian Knox wrote:
From your diagram, it looks like you are trying to load balance RELP. As
far as I know, RELP does not suppot ActionTCPRebindInterval. I believe
this has been discussed on the mailing list:
http://lists.adiscon.net/pipermail/rsyslog/2013-May/032549.html
Unless something has changed, you need to use the omfwd module if you
want
to use tcp rebinding. This isn't a bug - this is documented behavior.
The
rebind interval parameter is documented as a parameter for omfwd. RELP
uses omrelp, which has no such paramater. See:
http://www.rsyslog.com/doc/v8-stable/configuration/modules/omfwd.html
and
http://www.rsyslog.com/doc/v8-stable/configuration/modules/omrelp.html
with the new style config it does in the current git branch. It looks
like
it was added in 7.3.15
/* tables for interfacing with the v6 config system */
/* action (instance) parameters */
static struct cnfparamdescr actpdescr[] = {
{ target, eCmdHdlrGetWord, 1 },
{ tls, eCmdHdlrBinary, 0 },
{ tls.compression, eCmdHdlrBinary, 0 },
{ tls.prioritystring, eCmdHdlrString, 0 },
{ tls.cacert, eCmdHdlrString, 0 },
{ tls.mycert, eCmdHdlrString, 0 },
{ tls.myprivkey, eCmdHdlrString, 0 },
{ tls.authmode, eCmdHdlrString, 0 },
{ tls.permittedpeer, eCmdHdlrArray, 0 },
{ port, eCmdHdlrGetWord, 0 },
{ rebindinterval, eCmdHdlrInt, 0 },
{ windowsize, eCmdHdlrInt, 0 },
{ timeout, eCmdHdlrInt, 0 },
{ localclientip, eCmdHdlrGetWord, 0 },
{ template, eCmdHdlrGetWord, 0 }
};
I use ActionTCPRebindInterval with haproxy with plain TCP. It works
very
well.
Cheers,
Brian
On Thu, Jul 23, 2015 at 7:03 AM, smain...@free.fr wrote:
With the architecture enclosed.
- Mail original -
De: smain...@free.fr
À: rsyslog-users rsyslog@lists.adiscon.com
Envoyé: Jeudi 23 Juillet 2015 11:59:35
Objet: [rsyslog] Load balancing issue
Hello all,
I'm currently trying to load balance the log traffic accross several
servers.
I thought my configuration with ActionSendTCPRebindInterval option
was
working properly, unfortunately my recent benchs show that the log flow
is
not well load balanced.
Please find below a part of the architecture :
My problem is located on the log aggregators : the rsyslog send its
traffic to haproxy on localhost using relp protocol.
I monitored the tcp sessions and i can see that haproxy doesn't change
the
destination servers.
watch 'ss -lap -o state established \( dport = :20514 \)'
Recv-Q Send-Q Local Address:Port Peer Address:Port
1716 0 127.0.0.1:43652 127.0.0.1:20514 users:((rsyslogd,8409,88))
0 1138 10.17.252.4:58436 10.19.12.5:20514 timer:(on,196ms,0)
users:((haproxy,3922
,2))
1760 0 127.0.0.1:43650 127.0.0.1:20514 users:((rsyslogd,8409,22))
0 0 10.17.252.4:55583 10.19.12.6:20514 users:((haproxy,3922,10))
Please find enclosed my configuration.
NB :
- the source pid (rsyslog) never change as it is expected with
ActionSendTCPRebindInterval
- i mixed legacy and new syntaxe because of the following bug
https://github.com/rsyslog/rsyslog/issues/96
This bug is annoying and i didn't receive any update since about 4
months
Could you please help me ?
OS : debian7
rsyslog version : 8.10
Regards ,
Smana
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
Re: [rsyslog] Load balancing issue
Actually i found the parameter and i changed my configuration.
input(type=imtcp port=514 ruleset=forward
supportOctetCountedFraming=off)
But it still doesn't work as expected.
Indeed inside my ruleset i have this condition :
if $programname startswith 'Myapp.' then {
action( type=omrelp
...
When i disable octet-counting it seems that this condition is not reached.
My log format look like that :
20150115003549 server Myapp.sometag
{response:{status:206,duration:1,size:311557},some_other: 4242}
And i use loggen (from syslog-ng) for my benchs.
Note: when i change the condition to
if $fromhost-ip == '10.x.x.x' then {
...
it seems that the condition is reached and i still have a segfault
thanks,
Smana
- Mail original -
De: Rainer Gerhards rgerha...@hq.adiscon.com
À: David Lang da...@lang.hm
Cc: rsyslog-users rsyslog@lists.adiscon.com
Envoyé: Jeudi 23 Juillet 2015 15:13:24
Objet: Re: [rsyslog] Load balancing issue
2015-07-23 15:12 GMT+02:00 David Lang da...@lang.hm:
On Thu, 23 Jul 2015, smain...@free.fr wrote:
1- it solved my issue regarding the disk queue not created. i'll update
the github issue.
good.
2- i have a lot of errors like rsyslogd: Framing Error in received TCP
message: invalid octet count -1871509715. [v8.10.0]
this means that you are getting malformed data sent to you. Rsyslog
implements an extension to the syslog protocol where instead of each log
message being a string of text followed by a newline, the sending system can
send a number at the beginning (instead of PRI where PRI is the combined
facility/severity data) and rsyslog will then read that number of bytes as
the message. This allows a message to contain embedded newlines.
What's happening is that you have something sending you digits at the
beginning of the message, rsyslog is trying to interpret this, but it's
garbage data. I don't know if there is a way to disable octet counted mode
on the reciever or not.
There is a parameter to do that, but I don't remember the name out of my head.
Rainer
similarly, a message starting with 'z' is
interpreted as a compressed message.
The 'best' answer is to figure out which system is generating the invalid
messages and fix it there. If you can do so.
David Lang
3- As soon as i enable the rebindInterval option, rsyslog segfaults
[Thu Jul 23 12:46:03 2015] rs:analytics qu[19247]: segfault at 20 ip
7f3a64efa624 sp 7f3a5b1f5bc8 error 4 in
librelp.so.0.1.0[7f3a64eee000+11000]
Please find the startup debug logs here :
https://gist.github.com/Smana/21f1add821b91f1a0bc1
Regards,
Smana
- Mail original -
De: Brian Knox bk...@digitalocean.com
À: rsyslog-users rsyslog@lists.adiscon.com
Envoyé: Jeudi 23 Juillet 2015 14:17:05
Objet: Re: [rsyslog] Load balancing issue
Aha! David - to summarize, is the problem then that:
a) the parameter did not exist previously, and
b) was only added for the new style configs?
Brian
On Thu, Jul 23, 2015 at 7:59 AM, David Lang da...@lang.hm wrote:
On Thu, 23 Jul 2015, Brian Knox wrote:
From your diagram, it looks like you are trying to load balance RELP. As
far as I know, RELP does not suppot ActionTCPRebindInterval. I believe
this has been discussed on the mailing list:
http://lists.adiscon.net/pipermail/rsyslog/2013-May/032549.html
Unless something has changed, you need to use the omfwd module if you
want
to use tcp rebinding. This isn't a bug - this is documented behavior.
The
rebind interval parameter is documented as a parameter for omfwd. RELP
uses omrelp, which has no such paramater. See:
http://www.rsyslog.com/doc/v8-stable/configuration/modules/omfwd.html
and
http://www.rsyslog.com/doc/v8-stable/configuration/modules/omrelp.html
with the new style config it does in the current git branch. It looks
like
it was added in 7.3.15
/* tables for interfacing with the v6 config system */
/* action (instance) parameters */
static struct cnfparamdescr actpdescr[] = {
{ target, eCmdHdlrGetWord, 1 },
{ tls, eCmdHdlrBinary, 0 },
{ tls.compression, eCmdHdlrBinary, 0 },
{ tls.prioritystring, eCmdHdlrString, 0 },
{ tls.cacert, eCmdHdlrString, 0 },
{ tls.mycert, eCmdHdlrString, 0 },
{ tls.myprivkey, eCmdHdlrString, 0 },
{ tls.authmode, eCmdHdlrString, 0 },
{ tls.permittedpeer, eCmdHdlrArray, 0 },
{ port, eCmdHdlrGetWord, 0 },
{ rebindinterval, eCmdHdlrInt, 0 },
{ windowsize, eCmdHdlrInt, 0 },
{ timeout, eCmdHdlrInt, 0 },
{ localclientip, eCmdHdlrGetWord, 0 },
{ template, eCmdHdlrGetWord, 0 }
};
I use ActionTCPRebindInterval with haproxy with plain TCP. It works
very
well.
Cheers,
Brian
On Thu, Jul 23, 2015 at 7:03 AM, smain...@free.fr wrote:
With the architecture enclosed.
- Mail original -
De: smain...@free.fr
À: rsyslog-users rsyslog@lists.adiscon.com
Envoyé: Jeudi 23 Juillet 2015 11:59:35
Re: [rsyslog] Load balancing issue
So to resume there's 2 remaining issues i would like to address :
* disabling octet count causes that my condition based on appname to be ignored
* i have a segfault when i enable rebindInterval
Could you please tell me what are the information you need to help me ?
Thank you again,
Smana
- Mail original -
De: smain...@free.fr
À: rsyslog-users rsyslog@lists.adiscon.com
Envoyé: Jeudi 23 Juillet 2015 15:47:56
Objet: Re: [rsyslog] Load balancing issue
Actually i found the parameter and i changed my configuration.
input(type=imtcp port=514 ruleset=forward
supportOctetCountedFraming=off)
But it still doesn't work as expected.
Indeed inside my ruleset i have this condition :
if $programname startswith 'Myapp.' then {
action( type=omrelp
...
When i disable octet-counting it seems that this condition is not reached.
My log format look like that :
20150115003549 server Myapp.sometag
{response:{status:206,duration:1,size:311557},some_other: 4242}
And i use loggen (from syslog-ng) for my benchs.
Note: when i change the condition to
if $fromhost-ip == '10.x.x.x' then {
...
it seems that the condition is reached and i still have a segfault
thanks,
Smana
- Mail original -
De: Rainer Gerhards rgerha...@hq.adiscon.com
À: David Lang da...@lang.hm
Cc: rsyslog-users rsyslog@lists.adiscon.com
Envoyé: Jeudi 23 Juillet 2015 15:13:24
Objet: Re: [rsyslog] Load balancing issue
2015-07-23 15:12 GMT+02:00 David Lang da...@lang.hm:
On Thu, 23 Jul 2015, smain...@free.fr wrote:
1- it solved my issue regarding the disk queue not created. i'll update
the github issue.
good.
2- i have a lot of errors like rsyslogd: Framing Error in received TCP
message: invalid octet count -1871509715. [v8.10.0]
this means that you are getting malformed data sent to you. Rsyslog
implements an extension to the syslog protocol where instead of each log
message being a string of text followed by a newline, the sending system can
send a number at the beginning (instead of PRI where PRI is the combined
facility/severity data) and rsyslog will then read that number of bytes as
the message. This allows a message to contain embedded newlines.
What's happening is that you have something sending you digits at the
beginning of the message, rsyslog is trying to interpret this, but it's
garbage data. I don't know if there is a way to disable octet counted mode
on the reciever or not.
There is a parameter to do that, but I don't remember the name out of my head.
Rainer
similarly, a message starting with 'z' is
interpreted as a compressed message.
The 'best' answer is to figure out which system is generating the invalid
messages and fix it there. If you can do so.
David Lang
3- As soon as i enable the rebindInterval option, rsyslog segfaults
[Thu Jul 23 12:46:03 2015] rs:analytics qu[19247]: segfault at 20 ip
7f3a64efa624 sp 7f3a5b1f5bc8 error 4 in
librelp.so.0.1.0[7f3a64eee000+11000]
Please find the startup debug logs here :
https://gist.github.com/Smana/21f1add821b91f1a0bc1
Regards,
Smana
- Mail original -
De: Brian Knox bk...@digitalocean.com
À: rsyslog-users rsyslog@lists.adiscon.com
Envoyé: Jeudi 23 Juillet 2015 14:17:05
Objet: Re: [rsyslog] Load balancing issue
Aha! David - to summarize, is the problem then that:
a) the parameter did not exist previously, and
b) was only added for the new style configs?
Brian
On Thu, Jul 23, 2015 at 7:59 AM, David Lang da...@lang.hm wrote:
On Thu, 23 Jul 2015, Brian Knox wrote:
From your diagram, it looks like you are trying to load balance RELP. As
far as I know, RELP does not suppot ActionTCPRebindInterval. I believe
this has been discussed on the mailing list:
http://lists.adiscon.net/pipermail/rsyslog/2013-May/032549.html
Unless something has changed, you need to use the omfwd module if you
want
to use tcp rebinding. This isn't a bug - this is documented behavior.
The
rebind interval parameter is documented as a parameter for omfwd. RELP
uses omrelp, which has no such paramater. See:
http://www.rsyslog.com/doc/v8-stable/configuration/modules/omfwd.html
and
http://www.rsyslog.com/doc/v8-stable/configuration/modules/omrelp.html
with the new style config it does in the current git branch. It looks
like
it was added in 7.3.15
/* tables for interfacing with the v6 config system */
/* action (instance) parameters */
static struct cnfparamdescr actpdescr[] = {
{ target, eCmdHdlrGetWord, 1 },
{ tls, eCmdHdlrBinary, 0 },
{ tls.compression, eCmdHdlrBinary, 0 },
{ tls.prioritystring, eCmdHdlrString, 0 },
{ tls.cacert, eCmdHdlrString, 0 },
{ tls.mycert, eCmdHdlrString, 0 },
{ tls.myprivkey, eCmdHdlrString, 0 },
{ tls.authmode, eCmdHdlrString, 0 },
{ tls.permittedpeer, eCmdHdlrArray, 0 },
{ port, eCmdHdlrGetWord, 0 },
{ rebindinterval, eCmdHdlrInt, 0 },
{ windowsize, eCmdHdlrInt, 0
Re: [rsyslog] Load balancing issue
On Thu, 23 Jul 2015, smain...@free.fr wrote: Hello all, I'm currently trying to load balance the log traffic accross several servers. I thought my configuration with ActionSendTCPRebindInterval option was working properly, unfortunately my recent benchs show that the log flow is not well load balanced. Please find below a part of the architecture : My problem is located on the log aggregators : the rsyslog send its traffic to haproxy on localhost using relp protocol. I monitored the tcp sessions and i can see that haproxy doesn't change the destination servers. watch 'ss -lap -o state established \( dport = :20514 \)' Recv-Q Send-Q Local Address:Port Peer Address:Port 1716 0 127.0.0.1:43652 127.0.0.1:20514 users:((rsyslogd,8409,88)) 0 1138 10.17.252.4:58436 10.19.12.5:20514 timer:(on,196ms,0) users:((haproxy,3922 ,2)) 1760 0 127.0.0.1:43650 127.0.0.1:20514 users:((rsyslogd,8409,22)) 0 0 10.17.252.4:55583 10.19.12.6:20514 users:((haproxy,3922,10)) Please find enclosed my configuration. NB : - the source pid (rsyslog) never change as it is expected with ActionSendTCPRebindInterval the source pid would not change. the source port will change, but not the pid. how is haproxy configured? - i mixed legacy and new syntaxe because of the following bug https://github.com/rsyslog/rsyslog/issues/96 This bug is annoying and i didn't receive any update since about 4 months hmm, there is another thread in the last day about some queue parameters not being set by default, take a look at it and try setting those parameters (subject line Disk queue to flush after restart) reading the docs and code, it's not clear to me that omrelp accepts a rebindinteval from the old style config format (one problem with the old style format is that you can't know from reading the config what values will be used by the module. looking at the function INITLegCnfVars in omrelp.c it looks like it doesn't set or honor any legacy config parameters, so I don't think that you are getting teh rebindinterval set the way you think you are. try running rsyslog -dn and look for the config settings like the other thread I mentioned above does David Lang ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Load balancing issue
Aha! David - to summarize, is the problem then that:
a) the parameter did not exist previously, and
b) was only added for the new style configs?
Brian
On Thu, Jul 23, 2015 at 7:59 AM, David Lang da...@lang.hm wrote:
On Thu, 23 Jul 2015, Brian Knox wrote:
From your diagram, it looks like you are trying to load balance RELP. As
far as I know, RELP does not suppot ActionTCPRebindInterval. I believe
this has been discussed on the mailing list:
http://lists.adiscon.net/pipermail/rsyslog/2013-May/032549.html
Unless something has changed, you need to use the omfwd module if you want
to use tcp rebinding. This isn't a bug - this is documented behavior.
The
rebind interval parameter is documented as a parameter for omfwd. RELP
uses omrelp, which has no such paramater. See:
http://www.rsyslog.com/doc/v8-stable/configuration/modules/omfwd.html and
http://www.rsyslog.com/doc/v8-stable/configuration/modules/omrelp.html
with the new style config it does in the current git branch. It looks like
it was added in 7.3.15
/* tables for interfacing with the v6 config system */
/* action (instance) parameters */
static struct cnfparamdescr actpdescr[] = {
{ target, eCmdHdlrGetWord, 1 },
{ tls, eCmdHdlrBinary, 0 },
{ tls.compression, eCmdHdlrBinary, 0 },
{ tls.prioritystring, eCmdHdlrString, 0 },
{ tls.cacert, eCmdHdlrString, 0 },
{ tls.mycert, eCmdHdlrString, 0 },
{ tls.myprivkey, eCmdHdlrString, 0 },
{ tls.authmode, eCmdHdlrString, 0 },
{ tls.permittedpeer, eCmdHdlrArray, 0 },
{ port, eCmdHdlrGetWord, 0 },
{ rebindinterval, eCmdHdlrInt, 0 },
{ windowsize, eCmdHdlrInt, 0 },
{ timeout, eCmdHdlrInt, 0 },
{ localclientip, eCmdHdlrGetWord, 0 },
{ template, eCmdHdlrGetWord, 0 }
};
I use ActionTCPRebindInterval with haproxy with plain TCP. It works very
well.
Cheers,
Brian
On Thu, Jul 23, 2015 at 7:03 AM, smain...@free.fr wrote:
With the architecture enclosed.
- Mail original -
De: smain...@free.fr
À: rsyslog-users rsyslog@lists.adiscon.com
Envoyé: Jeudi 23 Juillet 2015 11:59:35
Objet: [rsyslog] Load balancing issue
Hello all,
I'm currently trying to load balance the log traffic accross several
servers.
I thought my configuration with ActionSendTCPRebindInterval option was
working properly, unfortunately my recent benchs show that the log flow
is
not well load balanced.
Please find below a part of the architecture :
My problem is located on the log aggregators : the rsyslog send its
traffic to haproxy on localhost using relp protocol.
I monitored the tcp sessions and i can see that haproxy doesn't change
the
destination servers.
watch 'ss -lap -o state established \( dport = :20514 \)'
Recv-Q Send-Q Local Address:Port Peer Address:Port
1716 0 127.0.0.1:43652 127.0.0.1:20514 users:((rsyslogd,8409,88))
0 1138 10.17.252.4:58436 10.19.12.5:20514 timer:(on,196ms,0)
users:((haproxy,3922
,2))
1760 0 127.0.0.1:43650 127.0.0.1:20514 users:((rsyslogd,8409,22))
0 0 10.17.252.4:55583 10.19.12.6:20514 users:((haproxy,3922,10))
Please find enclosed my configuration.
NB :
- the source pid (rsyslog) never change as it is expected with
ActionSendTCPRebindInterval
- i mixed legacy and new syntaxe because of the following bug
https://github.com/rsyslog/rsyslog/issues/96
This bug is annoying and i didn't receive any update since about 4 months
Could you please help me ?
OS : debian7
rsyslog version : 8.10
Regards ,
Smana
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com
Re: [rsyslog] Load balancing issue
On Thu, 23 Jul 2015, smain...@free.fr wrote:
1- it solved my issue regarding the disk queue not created. i'll update the
github issue.
good.
2- i have a lot of errors like
rsyslogd: Framing Error in received TCP message: invalid octet count -1871509715. [v8.10.0]
this means that you are getting malformed data sent to you. Rsyslog implements
an extension to the syslog protocol where instead of each log message being a
string of text followed by a newline, the sending system can send a number at
the beginning (instead of PRI where PRI is the combined facility/severity
data) and rsyslog will then read that number of bytes as the message. This
allows a message to contain embedded newlines.
What's happening is that you have something sending you digits at the beginning
of the message, rsyslog is trying to interpret this, but it's garbage data. I
don't know if there is a way to disable octet counted mode on the reciever or
not. similarly, a message starting with 'z' is interpreted as a compressed
message.
The 'best' answer is to figure out which system is generating the invalid
messages and fix it there. If you can do so.
David Lang
3- As soon as i enable the rebindInterval option, rsyslog segfaults
[Thu Jul 23 12:46:03 2015] rs:analytics qu[19247]: segfault at 20 ip
7f3a64efa624 sp 7f3a5b1f5bc8 error 4 in
librelp.so.0.1.0[7f3a64eee000+11000]
Please find the startup debug logs here :
https://gist.github.com/Smana/21f1add821b91f1a0bc1
Regards,
Smana
- Mail original -
De: Brian Knox bk...@digitalocean.com
À: rsyslog-users rsyslog@lists.adiscon.com
Envoyé: Jeudi 23 Juillet 2015 14:17:05
Objet: Re: [rsyslog] Load balancing issue
Aha! David - to summarize, is the problem then that:
a) the parameter did not exist previously, and
b) was only added for the new style configs?
Brian
On Thu, Jul 23, 2015 at 7:59 AM, David Lang da...@lang.hm wrote:
On Thu, 23 Jul 2015, Brian Knox wrote:
From your diagram, it looks like you are trying to load balance RELP. As
far as I know, RELP does not suppot ActionTCPRebindInterval. I believe
this has been discussed on the mailing list:
http://lists.adiscon.net/pipermail/rsyslog/2013-May/032549.html
Unless something has changed, you need to use the omfwd module if you want
to use tcp rebinding. This isn't a bug - this is documented behavior.
The
rebind interval parameter is documented as a parameter for omfwd. RELP
uses omrelp, which has no such paramater. See:
http://www.rsyslog.com/doc/v8-stable/configuration/modules/omfwd.html and
http://www.rsyslog.com/doc/v8-stable/configuration/modules/omrelp.html
with the new style config it does in the current git branch. It looks like
it was added in 7.3.15
/* tables for interfacing with the v6 config system */
/* action (instance) parameters */
static struct cnfparamdescr actpdescr[] = {
{ target, eCmdHdlrGetWord, 1 },
{ tls, eCmdHdlrBinary, 0 },
{ tls.compression, eCmdHdlrBinary, 0 },
{ tls.prioritystring, eCmdHdlrString, 0 },
{ tls.cacert, eCmdHdlrString, 0 },
{ tls.mycert, eCmdHdlrString, 0 },
{ tls.myprivkey, eCmdHdlrString, 0 },
{ tls.authmode, eCmdHdlrString, 0 },
{ tls.permittedpeer, eCmdHdlrArray, 0 },
{ port, eCmdHdlrGetWord, 0 },
{ rebindinterval, eCmdHdlrInt, 0 },
{ windowsize, eCmdHdlrInt, 0 },
{ timeout, eCmdHdlrInt, 0 },
{ localclientip, eCmdHdlrGetWord, 0 },
{ template, eCmdHdlrGetWord, 0 }
};
I use ActionTCPRebindInterval with haproxy with plain TCP. It works very
well.
Cheers,
Brian
On Thu, Jul 23, 2015 at 7:03 AM, smain...@free.fr wrote:
With the architecture enclosed.
- Mail original -
De: smain...@free.fr
À: rsyslog-users rsyslog@lists.adiscon.com
Envoyé: Jeudi 23 Juillet 2015 11:59:35
Objet: [rsyslog] Load balancing issue
Hello all,
I'm currently trying to load balance the log traffic accross several
servers.
I thought my configuration with ActionSendTCPRebindInterval option was
working properly, unfortunately my recent benchs show that the log flow
is
not well load balanced.
Please find below a part of the architecture :
My problem is located on the log aggregators : the rsyslog send its
traffic to haproxy on localhost using relp protocol.
I monitored the tcp sessions and i can see that haproxy doesn't change
the
destination servers.
watch 'ss -lap -o state established \( dport = :20514 \)'
Recv-Q Send-Q Local Address:Port Peer Address:Port
1716 0 127.0.0.1:43652 127.0.0.1:20514 users:((rsyslogd,8409,88))
0 1138 10.17.252.4:58436 10.19.12.5:20514 timer:(on,196ms,0)
users:((haproxy,3922
,2))
1760 0 127.0.0.1:43650 127.0.0.1:20514 users:((rsyslogd,8409,22))
0 0 10.17.252.4:55583 10.19.12.6:20514 users:((haproxy,3922,10))
Please find enclosed my configuration.
NB :
- the source pid (rsyslog) never change as it is expected with
ActionSendTCPRebindInterval
- i mixed legacy and new syntaxe because
Re: [rsyslog] Load balancing issue
On Thu, 23 Jul 2015, smain...@free.fr wrote: So to resume there's 2 remaining issues i would like to address : * disabling octet count causes that my condition based on appname to be ignored see my other message about fixing the format. * i have a segfault when i enable rebindInterval this looks like a bug that's been reported and someone has been assigned to work onit. I'd suggest watching this bug and posting if alorbach has any trouble duplicating things https://github.com/rsyslog/rsyslog/issues/120 In the meantime, as a poor-man's work-around, you can setup a script that does something like while true do killall -HUP rsyslogd sleep 10 done this will send the HUP signal to rsyslog every 10 seconds, which will cause it to close it's outputs (the same thing the rebindinterval does every X messages) David Lang___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Load balancing issue
On Thu, 23 Jul 2015, smain...@free.fr wrote:
Actually i found the parameter and i changed my configuration.
input(type=imtcp port=514 ruleset=forward
supportOctetCountedFraming=off)
But it still doesn't work as expected.
Indeed inside my ruleset i have this condition :
if $programname startswith 'Myapp.' then {
action( type=omrelp
...
When i disable octet-counting it seems that this condition is not reached.
My log format look like that :
20150115003549 server Myapp.sometag
{response:{status:206,duration:1,size:311557},some_other: 4242}
the problem is that this is not a valid syslog message as that's not a valid
timestamp format.
there are two legitimate formats you can use to send messages
15Jan 15 00:35:49 server Myapp.sometag: some message including json
1 152015-01-15T00:35:49-07:00 server Myapp.sometag: some message including
json
since you send 20150115003549 instead, rsyslog can't figure out what it is. It
was trying to interpret this as the octet framing, but luckily for you it's too
large a number to be legitimate (which is why you were getting the errors, but
were getting a usable message). Once you tell rsyslog that it's not octet
framing, rsyslog then guesses that it's the server name, which would make the
programname server
As I said before, the right way is to fix the sender that's sending the bad
format :-)
David Lang
And i use loggen (from syslog-ng) for my benchs.
Note: when i change the condition to
if $fromhost-ip == '10.x.x.x' then {
...
it seems that the condition is reached and i still have a segfault
thanks,
Smana
- Mail original -
De: Rainer Gerhards rgerha...@hq.adiscon.com
À: David Lang da...@lang.hm
Cc: rsyslog-users rsyslog@lists.adiscon.com
Envoyé: Jeudi 23 Juillet 2015 15:13:24
Objet: Re: [rsyslog] Load balancing issue
2015-07-23 15:12 GMT+02:00 David Lang da...@lang.hm:
On Thu, 23 Jul 2015, smain...@free.fr wrote:
1- it solved my issue regarding the disk queue not created. i'll update
the github issue.
good.
2- i have a lot of errors like rsyslogd: Framing Error in received TCP
message: invalid octet count -1871509715. [v8.10.0]
this means that you are getting malformed data sent to you. Rsyslog
implements an extension to the syslog protocol where instead of each log
message being a string of text followed by a newline, the sending system can
send a number at the beginning (instead of PRI where PRI is the combined
facility/severity data) and rsyslog will then read that number of bytes as
the message. This allows a message to contain embedded newlines.
What's happening is that you have something sending you digits at the
beginning of the message, rsyslog is trying to interpret this, but it's
garbage data. I don't know if there is a way to disable octet counted mode
on the reciever or not.
There is a parameter to do that, but I don't remember the name out of my head.
Rainer
similarly, a message starting with 'z' is
interpreted as a compressed message.
The 'best' answer is to figure out which system is generating the invalid
messages and fix it there. If you can do so.
David Lang
3- As soon as i enable the rebindInterval option, rsyslog segfaults
[Thu Jul 23 12:46:03 2015] rs:analytics qu[19247]: segfault at 20 ip
7f3a64efa624 sp 7f3a5b1f5bc8 error 4 in
librelp.so.0.1.0[7f3a64eee000+11000]
Please find the startup debug logs here :
https://gist.github.com/Smana/21f1add821b91f1a0bc1
Regards,
Smana
- Mail original -
De: Brian Knox bk...@digitalocean.com
À: rsyslog-users rsyslog@lists.adiscon.com
Envoyé: Jeudi 23 Juillet 2015 14:17:05
Objet: Re: [rsyslog] Load balancing issue
Aha! David - to summarize, is the problem then that:
a) the parameter did not exist previously, and
b) was only added for the new style configs?
Brian
On Thu, Jul 23, 2015 at 7:59 AM, David Lang da...@lang.hm wrote:
On Thu, 23 Jul 2015, Brian Knox wrote:
From your diagram, it looks like you are trying to load balance RELP. As
far as I know, RELP does not suppot ActionTCPRebindInterval. I believe
this has been discussed on the mailing list:
http://lists.adiscon.net/pipermail/rsyslog/2013-May/032549.html
Unless something has changed, you need to use the omfwd module if you
want
to use tcp rebinding. This isn't a bug - this is documented behavior.
The
rebind interval parameter is documented as a parameter for omfwd. RELP
uses omrelp, which has no such paramater. See:
http://www.rsyslog.com/doc/v8-stable/configuration/modules/omfwd.html
and
http://www.rsyslog.com/doc/v8-stable/configuration/modules/omrelp.html
with the new style config it does in the current git branch. It looks
like
it was added in 7.3.15
/* tables for interfacing with the v6 config system */
/* action (instance) parameters */
static struct cnfparamdescr actpdescr[] = {
{ target, eCmdHdlrGetWord, 1 },
{ tls, eCmdHdlrBinary, 0 },
{ tls.compression, eCmdHdlrBinary, 0 },
{ tls.prioritystring, eCmdHdlrString, 0
[rsyslog] Load balancing issue
Hello all,
I'm currently trying to load balance the log traffic accross several servers.
I thought my configuration with ActionSendTCPRebindInterval option was
working properly, unfortunately my recent benchs show that the log flow is not
well load balanced.
Please find below a part of the architecture :
My problem is located on the log aggregators : the rsyslog send its traffic to
haproxy on localhost using relp protocol.
I monitored the tcp sessions and i can see that haproxy doesn't change the
destination servers.
watch 'ss -lap -o state established \( dport = :20514 \)'
Recv-Q Send-Q Local Address:Port Peer Address:Port
1716 0 127.0.0.1:43652 127.0.0.1:20514 users:((rsyslogd,8409,88))
0 1138 10.17.252.4:58436 10.19.12.5:20514 timer:(on,196ms,0)
users:((haproxy,3922
,2))
1760 0 127.0.0.1:43650 127.0.0.1:20514 users:((rsyslogd,8409,22))
0 0 10.17.252.4:55583 10.19.12.6:20514 users:((haproxy,3922,10))
Please find enclosed my configuration.
NB :
- the source pid (rsyslog) never change as it is expected with
ActionSendTCPRebindInterval
- i mixed legacy and new syntaxe because of the following bug
https://github.com/rsyslog/rsyslog/issues/96
This bug is annoying and i didn't receive any update since about 4 months
Could you please help me ?
OS : debian7
rsyslog version : 8.10
Regards ,
Smana
module(
load=impstats
interval=30
resetCounters=off
format=cee
log.file=/var/log/rsyslog-stats.log
ruleset=monitoring
)
module(load=mmjsonparse)
module(load=omelasticsearch)
module(load=omrelp)
module(load=imtcp)
input(type=imtcp port=514 ruleset=forward)
template(name=stats
type=list) {
constant(value={)
property(name=timereported dateFormat=rfc3339 format=jsonf
outname=@timestamp) # the timestamp
constant(value=,)
property(name=hostname format=jsonf outname=host) # the host
generating stats
constant(value=,\source\:\impstats\,)
property(name=$!all-json position.from=2)
}
ruleset(
name=monitoring
queue.type=FixedArray
queue.highwatermark=18
queue.filename=stats_ruleset
queue.lowwatermark=2
queue.maxdiskspace=100m
queue.size=300
queue.dequeuebatchsize=1000
queue.saveonshutdown=on
){
action(
name=parse_impstats
type=mmjsonparse
)
action(
name=impstats_to_es
type=omelasticsearch
server=es-1
serverport=9200
template=stats
searchIndex=monitoring
searchType=rsyslog
bulkmode=on
action.resumeretrycount=-1
)
}
# Ruleset: Send logs to insight plateform
ruleset(name=forward){
$ActionName analytics
$ActionQueueType LinkedList
$ActionQueueFileName analytics-spool
$ActionResumeRetryCount -1
$ActionQueueSaveOnShutdown on
$ActionQueueHighWaterMark 1
$ActionSendTCPRebindInterval 1
if $programname startswith 'CDN.' then :omrelp:localhost:20514
}
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
Re: [rsyslog] Load balancing issue
With the architecture enclosed. - Mail original - De: smain...@free.fr À: rsyslog-users rsyslog@lists.adiscon.com Envoyé: Jeudi 23 Juillet 2015 11:59:35 Objet: [rsyslog] Load balancing issue Hello all, I'm currently trying to load balance the log traffic accross several servers. I thought my configuration with ActionSendTCPRebindInterval option was working properly, unfortunately my recent benchs show that the log flow is not well load balanced. Please find below a part of the architecture : My problem is located on the log aggregators : the rsyslog send its traffic to haproxy on localhost using relp protocol. I monitored the tcp sessions and i can see that haproxy doesn't change the destination servers. watch 'ss -lap -o state established \( dport = :20514 \)' Recv-Q Send-Q Local Address:Port Peer Address:Port 1716 0 127.0.0.1:43652 127.0.0.1:20514 users:((rsyslogd,8409,88)) 0 1138 10.17.252.4:58436 10.19.12.5:20514 timer:(on,196ms,0) users:((haproxy,3922 ,2)) 1760 0 127.0.0.1:43650 127.0.0.1:20514 users:((rsyslogd,8409,22)) 0 0 10.17.252.4:55583 10.19.12.6:20514 users:((haproxy,3922,10)) Please find enclosed my configuration. NB : - the source pid (rsyslog) never change as it is expected with ActionSendTCPRebindInterval - i mixed legacy and new syntaxe because of the following bug https://github.com/rsyslog/rsyslog/issues/96 This bug is annoying and i didn't receive any update since about 4 months Could you please help me ? OS : debian7 rsyslog version : 8.10 Regards , Smana ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
