[rsyslog] Spooling server per datacenter
Hi guys, Could you please help me to find out the proper configuration for the following use case ? * We have multiple datacenters * All our logs are sent to a central analytic platform * In each dc we'd like to have a spooling server which will keep to logs in case of network failure. * All the logs from the sources servers have to be sent to the spooling server (no spooling on source servers) * Relp if it's possible To summarize : source servers -> spooling server -> analytics plateform I tried to use relp but when the destination (analytics pf) is unreachable all the log flow slows down, even on source servers. With tcp the source server keeps to send but i don't see my spooling space growing. I presume i'm loosing data (i'll do further tests) When i use the option "action.resumeRetryCount="-1" when the destination is uncheachable the log flow stops completely... Here is my current configuration Source server: module(load="impstats" format="json" interval="60" log.syslog="off" log.file="/var/log/rsyslog-stats.log" severity="7") module(load="imtcp") input(type="imtcp" port="514") if $programname startswith 'foo.' then @@bar.domain.tld:514 Spooling server: module(load="imtcp") input(type="imtcp" port="514") module(load="impstats" format="json" interval="60" log.syslog="off" log.file="/var/log/rsyslog-stats.log" severity="7") if $programname startswith 'foo.' then { action(type="omfwd" action.resumeRetryCount="-1" name="spooling" target="analytics" port="514" protocol="tcp" queue.filename="eggforward" queue.spoolDirectory="/var/spool/rsyslog" queue.type="LinkedList") } Thanks for your help Smana ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Spooling server per datacenter
Please let me know i you need more info. OS : debian wheezy rsyslog version : 8.8.0.ad1-1 Regards, Smana - Mail original - De: smain...@free.fr À: "rsyslog-users" Envoyé: Mercredi 11 Mars 2015 09:44:45 Objet: [rsyslog] Spooling server per datacenter Hi guys, Could you please help me to find out the proper configuration for the following use case ? * We have multiple datacenters * All our logs are sent to a central analytic platform * In each dc we'd like to have a spooling server which will keep to logs in case of network failure. * All the logs from the sources servers have to be sent to the spooling server (no spooling on source servers) * Relp if it's possible To summarize : source servers -> spooling server -> analytics plateform I tried to use relp but when the destination (analytics pf) is unreachable all the log flow slows down, even on source servers. With tcp the source server keeps to send but i don't see my spooling space growing. I presume i'm loosing data (i'll do further tests) When i use the option "action.resumeRetryCount="-1" when the destination is uncheachable the log flow stops completely... Here is my current configuration Source server: module(load="impstats" format="json" interval="60" log.syslog="off" log.file="/var/log/rsyslog-stats.log" severity="7") module(load="imtcp") input(type="imtcp" port="514") if $programname startswith 'foo.' then @@bar.domain.tld:514 Spooling server: module(load="imtcp") input(type="imtcp" port="514") module(load="impstats" format="json" interval="60" log.syslog="off" log.file="/var/log/rsyslog-stats.log" severity="7") if $programname startswith 'foo.' then { action(type="omfwd" action.resumeRetryCount="-1" name="spooling" target="analytics" port="514" protocol="tcp" queue.filename="eggforward" queue.spoolDirectory="/var/spool/rsyslog" queue.type="LinkedList") } Thanks for your help Smana ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Spooling server per datacenter
here are some things to get you started. When I get to work today I can give you examples of my live configs. https://www.usenix.org/publications/login/august-2013-volume-38-number-4/enterprise-logging https://www.usenix.org/publications/login/october-2013-volume-38-number-5/log-filtering-rsyslog to handle the problem of network interruptions backing things up, you will need to create some additional queues (lookup action queues and rulesets). I'll post more later. You are on the right track. David Lang On Wed, 11 Mar 2015, smain...@free.fr wrote: Date: Wed, 11 Mar 2015 15:37:19 +0100 (CET) From: smain...@free.fr Reply-To: rsyslog-users To: rsyslog-users Subject: Re: [rsyslog] Spooling server per datacenter Please let me know i you need more info. OS : debian wheezy rsyslog version : 8.8.0.ad1-1 Regards, Smana - Mail original - De: smain...@free.fr À: "rsyslog-users" Envoyé: Mercredi 11 Mars 2015 09:44:45 Objet: [rsyslog] Spooling server per datacenter Hi guys, Could you please help me to find out the proper configuration for the following use case ? * We have multiple datacenters * All our logs are sent to a central analytic platform * In each dc we'd like to have a spooling server which will keep to logs in case of network failure. * All the logs from the sources servers have to be sent to the spooling server (no spooling on source servers) * Relp if it's possible To summarize : source servers -> spooling server -> analytics plateform I tried to use relp but when the destination (analytics pf) is unreachable all the log flow slows down, even on source servers. With tcp the source server keeps to send but i don't see my spooling space growing. I presume i'm loosing data (i'll do further tests) When i use the option "action.resumeRetryCount="-1" when the destination is uncheachable the log flow stops completely... Here is my current configuration Source server: module(load="impstats" format="json" interval="60" log.syslog="off" log.file="/var/log/rsyslog-stats.log" severity="7") module(load="imtcp") input(type="imtcp" port="514") if $programname startswith 'foo.' then @@bar.domain.tld:514 Spooling server: module(load="imtcp") input(type="imtcp" port="514") module(load="impstats" format="json" interval="60" log.syslog="off" log.file="/var/log/rsyslog-stats.log" severity="7") if $programname startswith 'foo.' then { action(type="omfwd" action.resumeRetryCount="-1" name="spooling" target="analytics" port="514" protocol="tcp" queue.filename="eggforward" queue.spoolDirectory="/var/spool/rsyslog" queue.type="LinkedList") } Thanks for your help Smana ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Spooling server per datacenter
Hi David, Thanks for your answer. Waiting to hear you again :) Actually my main issue is to avoid to spool on the source server and send all my logs to the spooling server. Regards, Smana - Mail original - De: "David Lang" À: "rsyslog-users" Envoyé: Mercredi 11 Mars 2015 15:45:22 Objet: Re: [rsyslog] Spooling server per datacenter here are some things to get you started. When I get to work today I can give you examples of my live configs. https://www.usenix.org/publications/login/august-2013-volume-38-number-4/enterprise-logging https://www.usenix.org/publications/login/october-2013-volume-38-number-5/log-filtering-rsyslog to handle the problem of network interruptions backing things up, you will need to create some additional queues (lookup action queues and rulesets). I'll post more later. You are on the right track. David Lang On Wed, 11 Mar 2015, smain...@free.fr wrote: > Date: Wed, 11 Mar 2015 15:37:19 +0100 (CET) > From: smain...@free.fr > Reply-To: rsyslog-users > To: rsyslog-users > Subject: Re: [rsyslog] Spooling server per datacenter > > Please let me know i you need more info. > > OS : debian wheezy > rsyslog version : 8.8.0.ad1-1 > > Regards, > Smana > > > - Mail original - > De: smain...@free.fr > À: "rsyslog-users" > Envoyé: Mercredi 11 Mars 2015 09:44:45 > Objet: [rsyslog] Spooling server per datacenter > > Hi guys, > > Could you please help me to find out the proper configuration for the > following use case ? > > * We have multiple datacenters > * All our logs are sent to a central analytic platform > * In each dc we'd like to have a spooling server which will keep to logs in > case of network failure. > * All the logs from the sources servers have to be sent to the spooling > server (no spooling on source servers) > * Relp if it's possible > > To summarize : > source servers -> spooling server -> analytics plateform > > I tried to use relp but when the destination (analytics pf) is unreachable > all the log flow slows down, even on source servers. > With tcp the source server keeps to send but i don't see my spooling space > growing. I presume i'm loosing data (i'll do further tests) > When i use the option "action.resumeRetryCount="-1" when the destination is > uncheachable the log flow stops completely... > > Here is my current configuration > > Source server: > module(load="impstats" > format="json" > interval="60" > log.syslog="off" > log.file="/var/log/rsyslog-stats.log" > severity="7") > > module(load="imtcp") > input(type="imtcp" port="514") > > if $programname startswith 'foo.' then @@bar.domain.tld:514 > > Spooling server: > > module(load="imtcp") > input(type="imtcp" port="514") > > module(load="impstats" > format="json" > interval="60" > log.syslog="off" > log.file="/var/log/rsyslog-stats.log" > severity="7") > > if $programname startswith 'foo.' then { > action(type="omfwd" > action.resumeRetryCount="-1" > name="spooling" > target="analytics" > port="514" > protocol="tcp" > queue.filename="eggforward" > queue.spoolDirectory="/var/spool/rsyslog" > queue.type="LinkedList") > } > > Thanks for your help > Smana > ___ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. > ___ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ___ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
Re: [rsyslog] Spooling server per datacenter
Furthermore, i noticed some data loss when i use tcp without relp. In my use case we would like to avoid that if it's possible. Regards, Smana - Mail original - De: smain...@free.fr À: "rsyslog-users" Envoyé: Mercredi 11 Mars 2015 17:54:57 Objet: Re: [rsyslog] Spooling server per datacenter Hi David, Thanks for your answer. Waiting to hear you again :) Actually my main issue is to avoid to spool on the source server and send all my logs to the spooling server. Regards, Smana - Mail original - De: "David Lang" À: "rsyslog-users" Envoyé: Mercredi 11 Mars 2015 15:45:22 Objet: Re: [rsyslog] Spooling server per datacenter here are some things to get you started. When I get to work today I can give you examples of my live configs. https://www.usenix.org/publications/login/august-2013-volume-38-number-4/enterprise-logging https://www.usenix.org/publications/login/october-2013-volume-38-number-5/log-filtering-rsyslog to handle the problem of network interruptions backing things up, you will need to create some additional queues (lookup action queues and rulesets). I'll post more later. You are on the right track. David Lang On Wed, 11 Mar 2015, smain...@free.fr wrote: > Date: Wed, 11 Mar 2015 15:37:19 +0100 (CET) > From: smain...@free.fr > Reply-To: rsyslog-users > To: rsyslog-users > Subject: Re: [rsyslog] Spooling server per datacenter > > Please let me know i you need more info. > > OS : debian wheezy > rsyslog version : 8.8.0.ad1-1 > > Regards, > Smana > > > - Mail original - > De: smain...@free.fr > À: "rsyslog-users" > Envoyé: Mercredi 11 Mars 2015 09:44:45 > Objet: [rsyslog] Spooling server per datacenter > > Hi guys, > > Could you please help me to find out the proper configuration for the > following use case ? > > * We have multiple datacenters > * All our logs are sent to a central analytic platform > * In each dc we'd like to have a spooling server which will keep to logs in > case of network failure. > * All the logs from the sources servers have to be sent to the spooling > server (no spooling on source servers) > * Relp if it's possible > > To summarize : > source servers -> spooling server -> analytics plateform > > I tried to use relp but when the destination (analytics pf) is unreachable > all the log flow slows down, even on source servers. > With tcp the source server keeps to send but i don't see my spooling space > growing. I presume i'm loosing data (i'll do further tests) > When i use the option "action.resumeRetryCount="-1" when the destination is > uncheachable the log flow stops completely... > > Here is my current configuration > > Source server: > module(load="impstats" > format="json" > interval="60" > log.syslog="off" > log.file="/var/log/rsyslog-stats.log" > severity="7") > > module(load="imtcp") > input(type="imtcp" port="514") > > if $programname startswith 'foo.' then @@bar.domain.tld:514 > > Spooling server: > > module(load="imtcp") > input(type="imtcp" port="514") > > module(load="impstats" > format="json" > interval="60" > log.syslog="off" > log.file="/var/log/rsyslog-stats.log" > severity="7") > > if $programname startswith 'foo.' then { > action(type="omfwd" > action.resumeRetryCount="-1" > name="spooling" > target="analytics" > port="514" > protocol="tcp" > queue.filename="eggforward" > queue.spoolDirectory="/var/spool/rsyslog" > queue.type="LinkedList") > } > > Thanks for your help > Smana > ___ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. > ___ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO N
Re: [rsyslog] Spooling server per datacenter
ot;pstats_local") action(name="send_HP" type="omfwd" target="10.1.0.1" port="514" protocol="udp" template="structured_forwarding" queue.type="FixedArray") } # parse JSON messages to variables action(type="mmjsonparse") # if the message we got was in JSON from the beginning, there won't be a $!msg variable if $!msg == "" then set $!msg = $msg; # parse the origional message to extract fields set $.stdmsg = exec_template("stdmsg"); action(type="mmnormalize" path="$!extracted" variable="$.stdmsg" ruleBase="/root/rsyslog.rulebase") # if the message was extracted add timestamp and hostname as part of the # extracted data if $!extracted!originalmsg == '' then { set $!extracted!timestamp = $timestamp; set $!extracted!hostname = $hostname; } # if we failed to parse a cisco message, log what we failed to parse if $!extracted!originalmsg != '' and $programname startswith '%ASA-' then { /var/log/cisco-unknown-messages;unknown } # if this is a local log, send it to an edge relay. We do this so that it will # end up being delivered to all destinations. if $fromhost-ip == "127.0.0.1" then { if $programname startswith 'rsyslogd' then { /var/log/rsyslog-local-messages } set $!trusted!local!input = $inputname; @10.1.0.1 stop } # write logs in the traditional format without metadata /var/log/messages;std # write messages with full metadata and high precision timestamp /var/log/messages-full # forward messages to something that understands the JSON format and can use the metadata action(type="omfwd" name="smart-out" target="10.1.1.2" port="514" protocol="udp" template="structured_forwarding") # forward messages to something that only understands the traditional format action(type="omfwd" name="legacy-out" target="10.1.1.3" port="514" protocol="udp" template="std-fwd") David Lang Regards, Smana - Mail original - De: "David Lang" À: "rsyslog-users" Envoyé: Mercredi 11 Mars 2015 15:45:22 Objet: Re: [rsyslog] Spooling server per datacenter here are some things to get you started. When I get to work today I can give you examples of my live configs. https://www.usenix.org/publications/login/august-2013-volume-38-number-4/enterprise-logging https://www.usenix.org/publications/login/october-2013-volume-38-number-5/log-filtering-rsyslog to handle the problem of network interruptions backing things up, you will need to create some additional queues (lookup action queues and rulesets). I'll post more later. You are on the right track. David Lang On Wed, 11 Mar 2015, smain...@free.fr wrote: Date: Wed, 11 Mar 2015 15:37:19 +0100 (CET) From: smain...@free.fr Reply-To: rsyslog-users To: rsyslog-users Subject: Re: [rsyslog] Spooling server per datacenter Please let me know i you need more info. OS : debian wheezy rsyslog version : 8.8.0.ad1-1 Regards, Smana - Mail original - De: smain...@free.fr À: "rsyslog-users" Envoyé: Mercredi 11 Mars 2015 09:44:45 Objet: [rsyslog] Spooling server per datacenter Hi guys, Could you please help me to find out the proper configuration for the following use case ? * We have multiple datacenters * All our logs are sent to a central analytic platform * In each dc we'd like to have a spooling server which will keep to logs in case of network failure. * All the logs from the sources servers have to be sent to the spooling server (no spooling on source servers) * Relp if it's possible To summarize : source servers -> spooling server -> analytics plateform I tried to use relp but when the destination (analytics pf) is unreachable all the log flow slows down, even on source servers. With tcp the source server keeps to send but i don't see my spooling space growing. I presume i'm loosing data (i'll do further tests) When i use the option "action.resumeRetryCount="-1" when the destination is uncheachable the log flow stops completely... Here is my current configuration Source server: module(load="impstats" format="json" interval="60" log.syslog="off" log.file="/var/log/rsyslog-stats.log" severity="7") module(load="imtcp") input(type="imtcp" port="514") if $programname startswith 'foo.' then @@bar.domain.tld:514 Spooling server: module(load="imtcp") input(type="imtcp" port="514") module(load="impstats" format="json" interval="60" log.syslog="off" log.file="/var/log/rsyslog-stats.log" sev
Re: [rsyslog] Spooling server per datacenter
Hello David, Thank you for this example. Actually, i think i've found the desired configuration. Please let me know if you see any corner case. For reminder, i need to : * Avoid data loss * Spool on a single host The setup can be summarized as follows : Source server -> TCP -> Spooling server (DA queue) -> RELP -> Analytics Platform I will change the statistics configuration with a distinct ruleset soon. I've added a small queue as you suggested on the source server. Source server: -- module(load="impstats" format="json" interval="60" log.syslog="off" log.file="/var/log/rsyslog-stats.log" severity="7") module(load="imtcp") input(type="imtcp" port="514") if $programname startswith 'foo.' then { action(type="omfwd" action.resumeRetryCount="-1" target="syslog.domain.tld" port="514" protocol="tcp" queue.filename="cdnforward" queue.maxdiskspace="1g" queue.saveonshutdown="on" queue.spoolDirectory="/var/spool/rsyslog" queue.type="LinkedList") } Spooling server module(load="imtcp") input(type="imtcp" port="514") module(load="impstats" format="json" interval="60" log.syslog="off" log.file="/var/log/rsyslog-stats.log" severity="7") module(load="omrelp") if $programname startswith 'foo.' then { action(name="analytics" type="omrelp" target="analytics" port="20514" queue.fileName="analytics-buffer" queue.saveonshutdown="on" queue.type="LinkedList" action.resumeinterval="30" timeout="5") } Regards, Smana - Mail original - De: "David Lang" À: "rsyslog-users" Envoyé: Mercredi 11 Mars 2015 19:11:13 Objet: Re: [rsyslog] Spooling server per datacenter On Wed, 11 Mar 2015, smain...@free.fr wrote: > Hi David, > > Thanks for your answer. > Waiting to hear you again :) > > Actually my main issue is to avoid to spool on the source server and send all > my logs to the spooling server. The key question you need to think about is: When things go badly wrong with logging long enough, which would you rather have happen, loose logs or have your servers and applications stop? If you are not willing to loose logs, and don't want things queueing on the servers generating the logs, then you need to make the systems you are sending to redundant with auto-failover, and even then you are going to get some short delays. You really do want to have some spooling on the client sending to your local server, but you don't need a lot. I like to put a syslog relay/spooling server on each subnet so that there are no firewalls or ACLs between the systems generating the logs and the relay/spooling boxes. In this situation, simple UDP communications is very reliable (no bottlenecks where UDP is going to be at risk), and then I use TCP or RELP to relay from there to my central systems. what I do on my relay boxes is currently: # gather stats every 10 min. Process them independently of normal logs so that # if the normal log flow gets backed up, these stats will not be affected module(load="impstats" interval="600" resetCounters="on" format="legacy" ruleset="high_p") module(load="imuxsock" SysSock.Annotate="on" SysSock.ParseTrusted="on") module(load="imklog") module(load="imudp") input(type="imudp" port="514") module(load="imtcp" MaxSessions="1000") input(type="imtcp" port="514") module(load="mmjsonparse") action(type="mmjsonparse") if $fromhost-ip != "127.0.0.1" then { # if the log is being received from another machine, # add metadata to the log set $!trusted!origserver = $fromhost-ip; set $!trusted!edge!time = $timegenerated; set $!trusted!edge!relay = $$myhostname; set $!trusted!edge!input = $inputname; } else { set $!trusted!local!input = $inputname; } set $!trusted!environment = "Dev network"; $template structured_forwarding,"<%pri%>%timereported% %hostname% %syslogtag% @cee:%$!%\n" action(type="omfile" File="/var/log/messages" name="local_messages") action(type="omfile" File="/var/log/messages-full" template="structured_forwarding" name="cee_messages") a
Re: [rsyslog] Spooling server per datacenter
Just to keep you updated, i faced a problem described there : https://github.com/rsyslog/rsyslog/issues/96 I need to use the legacy format in order to spool properly. Regards, Smana - Mail original - De: smain...@free.fr À: "rsyslog-users" Envoyé: Jeudi 12 Mars 2015 14:22:39 Objet: Re: [rsyslog] Spooling server per datacenter Hello David, Thank you for this example. Actually, i think i've found the desired configuration. Please let me know if you see any corner case. For reminder, i need to : * Avoid data loss * Spool on a single host The setup can be summarized as follows : Source server -> TCP -> Spooling server (DA queue) -> RELP -> Analytics Platform I will change the statistics configuration with a distinct ruleset soon. I've added a small queue as you suggested on the source server. Source server: -- module(load="impstats" format="json" interval="60" log.syslog="off" log.file="/var/log/rsyslog-stats.log" severity="7") module(load="imtcp") input(type="imtcp" port="514") if $programname startswith 'foo.' then { action(type="omfwd" action.resumeRetryCount="-1" target="syslog.domain.tld" port="514" protocol="tcp" queue.filename="cdnforward" queue.maxdiskspace="1g" queue.saveonshutdown="on" queue.spoolDirectory="/var/spool/rsyslog" queue.type="LinkedList") } Spooling server module(load="imtcp") input(type="imtcp" port="514") module(load="impstats" format="json" interval="60" log.syslog="off" log.file="/var/log/rsyslog-stats.log" severity="7") module(load="omrelp") if $programname startswith 'foo.' then { action(name="analytics" type="omrelp" target="analytics" port="20514" queue.fileName="analytics-buffer" queue.saveonshutdown="on" queue.type="LinkedList" action.resumeinterval="30" timeout="5") } Regards, Smana - Mail original - De: "David Lang" À: "rsyslog-users" Envoyé: Mercredi 11 Mars 2015 19:11:13 Objet: Re: [rsyslog] Spooling server per datacenter On Wed, 11 Mar 2015, smain...@free.fr wrote: > Hi David, > > Thanks for your answer. > Waiting to hear you again :) > > Actually my main issue is to avoid to spool on the source server and send all > my logs to the spooling server. The key question you need to think about is: When things go badly wrong with logging long enough, which would you rather have happen, loose logs or have your servers and applications stop? If you are not willing to loose logs, and don't want things queueing on the servers generating the logs, then you need to make the systems you are sending to redundant with auto-failover, and even then you are going to get some short delays. You really do want to have some spooling on the client sending to your local server, but you don't need a lot. I like to put a syslog relay/spooling server on each subnet so that there are no firewalls or ACLs between the systems generating the logs and the relay/spooling boxes. In this situation, simple UDP communications is very reliable (no bottlenecks where UDP is going to be at risk), and then I use TCP or RELP to relay from there to my central systems. what I do on my relay boxes is currently: # gather stats every 10 min. Process them independently of normal logs so that # if the normal log flow gets backed up, these stats will not be affected module(load="impstats" interval="600" resetCounters="on" format="legacy" ruleset="high_p") module(load="imuxsock" SysSock.Annotate="on" SysSock.ParseTrusted="on") module(load="imklog") module(load="imudp") input(type="imudp" port="514") module(load="imtcp" MaxSessions="1000") input(type="imtcp" port="514") module(load="mmjsonparse") action(type="mmjsonparse") if $fromhost-ip != "127.0.0.1" then { # if the log is being received from another machine, # add metadata to the log set $!trusted!origserver = $fromhost-ip; set $!trusted!edge!time = $timegenerated; set $!trusted!edge!relay = $$myhostname; set $!trusted!edge!input = $inputname; } else { set $!trusted!local!input = $inputname; } set $!trusted!environment = "Dev network"; $template stru