Hello David,
Thank you for this example.
Actually, i think i've found the desired configuration.
Please let me know if you see any corner case.
For reminder, i need to :
* Avoid data loss
* Spool on a single host
The setup can be summarized as follows :
Source server -> TCP -> Spooling server (DA queue) -> RELP -> Analytics Platform
I will change the statistics configuration with a distinct ruleset soon.
I've added a small queue as you suggested on the source server.
Source server:
--------------
module(load="impstats"
format="json"
interval="60"
log.syslog="off"
log.file="/var/log/rsyslog-stats.log"
severity="7")
module(load="imtcp")
input(type="imtcp" port="514")
if $programname startswith 'foo.' then {
action(type="omfwd"
action.resumeRetryCount="-1"
target="syslog.domain.tld"
port="514"
protocol="tcp"
queue.filename="cdnforward"
queue.maxdiskspace="1g"
queue.saveonshutdown="on"
queue.spoolDirectory="/var/spool/rsyslog"
queue.type="LinkedList")
}
Spooling server
----------------
module(load="imtcp")
input(type="imtcp" port="514")
module(load="impstats"
format="json"
interval="60"
log.syslog="off"
log.file="/var/log/rsyslog-stats.log"
severity="7")
module(load="omrelp")
if $programname startswith 'foo.' then {
action(name="analytics"
type="omrelp"
target="analytics"
port="20514"
queue.fileName="analytics-buffer"
queue.saveonshutdown="on"
queue.type="LinkedList"
action.resumeinterval="30"
timeout="5")
}
Regards,
Smana
----- Mail original -----
De: "David Lang" <da...@lang.hm>
À: "rsyslog-users" <rsyslog@lists.adiscon.com>
Envoyé: Mercredi 11 Mars 2015 19:11:13
Objet: Re: [rsyslog] Spooling server per datacenter
On Wed, 11 Mar 2015, smain...@free.fr wrote:
> Hi David,
>
> Thanks for your answer.
> Waiting to hear you again :)
>
> Actually my main issue is to avoid to spool on the source server and send all
> my logs to the spooling server.
The key question you need to think about is:
When things go badly wrong with logging long enough, which would you rather
have
happen, loose logs or have your servers and applications stop?
If you are not willing to loose logs, and don't want things queueing on the
servers generating the logs, then you need to make the systems you are sending
to redundant with auto-failover, and even then you are going to get some short
delays.
You really do want to have some spooling on the client sending to your local
server, but you don't need a lot.
I like to put a syslog relay/spooling server on each subnet so that there are
no
firewalls or ACLs between the systems generating the logs and the
relay/spooling
boxes. In this situation, simple UDP communications is very reliable (no
bottlenecks where UDP is going to be at risk), and then I use TCP or RELP to
relay from there to my central systems.
what I do on my relay boxes is currently:
# gather stats every 10 min. Process them independently of normal logs so that
# if the normal log flow gets backed up, these stats will not be affected
module(load="impstats" interval="600" resetCounters="on" format="legacy"
ruleset="high_p")
module(load="imuxsock" SysSock.Annotate="on" SysSock.ParseTrusted="on")
module(load="imklog")
module(load="imudp")
input(type="imudp" port="514")
module(load="imtcp" MaxSessions="1000")
input(type="imtcp" port="514")
module(load="mmjsonparse")
action(type="mmjsonparse")
if $fromhost-ip != "127.0.0.1" then {
# if the log is being received from another machine,
# add metadata to the log
set $!trusted!origserver = $fromhost-ip;
set $!trusted!edge!time = $timegenerated;
set $!trusted!edge!relay = $$myhostname;
set $!trusted!edge!input = $inputname;
} else {
set $!trusted!local!input = $inputname;
}
set $!trusted!environment = "Dev network";
$template structured_forwarding,"<%pri%>%timereported% %hostname% %syslogtag%
@cee:%$!%\n"
action(type="omfile" File="/var/log/messages" name="local_messages")
action(type="omfile" File="/var/log/messages-full"
template="structured_forwarding" name="cee_messages")
action(type="omfwd" Target="10.1.1.1" Port="514" Protocol="tcp"
queue.type="FixedArray" template="structured_forwarding" name="send_remote")
# for high priority messages (the stats) write them locally and send them to the
# central server. Define queues for the ruleset and for the remote send to
# decouple them from being affected or affecting other logs
ruleset(name="high_p" queue.type="FixedArray"){
set $!trusted!local!input = $inputname;
action(type="mmjsonparse")
action(type="omfile" file="/var/log/pstats" name="pstats_local")
action(name="send_HP" type="omfwd" target="10.1.50.85" port="514"
protocol="tcp" queue.type="FixedArray" template="structured_forwarding")
}
On my central server I do:
module(load="impstats" interval="600" resetCounters="on" format="legacy"
ruleset="high_p")
module(load="imuxsock" SysSock.Annotate="on" SysSock.ParseTrusted="on")
module(load="imklog")
module(load="imtcp" MaxSessions="1000")
module(load="imudp" timerequery="4" )
module(load="mmnormalize")
input(type="imtcp" port="514")
input(type="imudp" port="514")
module(load="mmjsonparse")
# define templates
# send JSON message
$template structured_forwarding,"<%pri%>%timereported% %hostname% %syslogtag%
@cee:%$!%\n"
# local traditional format
$template stdmsg,"%timereported% %hostname% %syslogtag% %$!msg%"
$template std,"%$.stdmsg%\n"
$template unknown,"%$!extracted!originalmsg%\n"
# forward traditional format
$template std-fwd,"<%pri%>%timereported% %hostname% %syslogtag% %$!msg%\n"
# define a high priority queue that will send impstats immediately rather
# than going into the main queue and possibly being delayed by other logs
ruleset(name="high_p" queue.type="FixedArray"){
set $!trusted!local!input = $inputname;
action(type="mmjsonparse")
action(type="omfile" file="/var/log/pstats-local-messages"
name="pstats_local")
action(name="send_HP" type="omfwd" target="10.1.0.1" port="514"
protocol="udp" template="structured_forwarding" queue.type="FixedArray")
}
# parse JSON messages to variables
action(type="mmjsonparse")
# if the message we got was in JSON from the beginning, there won't be a $!msg
variable
if $!msg == "" then set $!msg = $msg;
# parse the origional message to extract fields
set $.stdmsg = exec_template("stdmsg");
action(type="mmnormalize" path="$!extracted" variable="$.stdmsg"
ruleBase="/root/rsyslog.rulebase")
# if the message was extracted add timestamp and hostname as part of the
# extracted data
if $!extracted!originalmsg == '' then {
set $!extracted!timestamp = $timestamp;
set $!extracted!hostname = $hostname;
}
# if we failed to parse a cisco message, log what we failed to parse
if $!extracted!originalmsg != '' and $programname startswith '%ASA-' then {
/var/log/cisco-unknown-messages;unknown
}
# if this is a local log, send it to an edge relay. We do this so that it will
# end up being delivered to all destinations.
if $fromhost-ip == "127.0.0.1" then {
if $programname startswith 'rsyslogd' then {
/var/log/rsyslog-local-messages
}
set $!trusted!local!input = $inputname;
@10.1.0.1
stop
}
# write logs in the traditional format without metadata
/var/log/messages;std
# write messages with full metadata and high precision timestamp
/var/log/messages-full
# forward messages to something that understands the JSON format and can use
the metadata
action(type="omfwd" name="smart-out" target="10.1.1.2" port="514"
protocol="udp" template="structured_forwarding")
# forward messages to something that only understands the traditional format
action(type="omfwd" name="legacy-out" target="10.1.1.3" port="514"
protocol="udp" template="std-fwd")
David Lang
>
> Regards,
> Smana
>
> ----- Mail original -----
> De: "David Lang" <da...@lang.hm>
> À: "rsyslog-users" <rsyslog@lists.adiscon.com>
> Envoyé: Mercredi 11 Mars 2015 15:45:22
> Objet: Re: [rsyslog] Spooling server per datacenter
>
> here are some things to get you started. When I get to work today I can give
> you
> examples of my live configs.
>
> https://www.usenix.org/publications/login/august-2013-volume-38-number-4/enterprise-logging
> https://www.usenix.org/publications/login/october-2013-volume-38-number-5/log-filtering-rsyslog
>
> to handle the problem of network interruptions backing things up, you will
> need
> to create some additional queues (lookup action queues and rulesets). I'll
> post
> more later.
>
> You are on the right track.
>
> David Lang
>
> On Wed, 11 Mar 2015, smain...@free.fr wrote:
>
>> Date: Wed, 11 Mar 2015 15:37:19 +0100 (CET)
>> From: smain...@free.fr
>> Reply-To: rsyslog-users <rsyslog@lists.adiscon.com>
>> To: rsyslog-users <rsyslog@lists.adiscon.com>
>> Subject: Re: [rsyslog] Spooling server per datacenter
>>
>> Please let me know i you need more info.
>>
>> OS : debian wheezy
>> rsyslog version : 8.8.0.ad1-1
>>
>> Regards,
>> Smana
>>
>>
>> ----- Mail original -----
>> De: smain...@free.fr
>> À: "rsyslog-users" <rsyslog@lists.adiscon.com>
>> Envoyé: Mercredi 11 Mars 2015 09:44:45
>> Objet: [rsyslog] Spooling server per datacenter
>>
>> Hi guys,
>>
>> Could you please help me to find out the proper configuration for the
>> following use case ?
>>
>> * We have multiple datacenters
>> * All our logs are sent to a central analytic platform
>> * In each dc we'd like to have a spooling server which will keep to logs in
>> case of network failure.
>> * All the logs from the sources servers have to be sent to the spooling
>> server (no spooling on source servers)
>> * Relp if it's possible
>>
>> To summarize :
>> source servers -> spooling server -> analytics plateform
>>
>> I tried to use relp but when the destination (analytics pf) is unreachable
>> all the log flow slows down, even on source servers.
>> With tcp the source server keeps to send but i don't see my spooling space
>> growing. I presume i'm loosing data (i'll do further tests)
>> When i use the option "action.resumeRetryCount="-1" when the destination is
>> uncheachable the log flow stops completely...
>>
>> Here is my current configuration
>>
>> Source server:
>> module(load="impstats"
>> format="json"
>> interval="60"
>> log.syslog="off"
>> log.file="/var/log/rsyslog-stats.log"
>> severity="7")
>>
>> module(load="imtcp")
>> input(type="imtcp" port="514")
>>
>> if $programname startswith 'foo.' then @@bar.domain.tld:514
>>
>> Spooling server:
>>
>> module(load="imtcp")
>> input(type="imtcp" port="514")
>>
>> module(load="impstats"
>> format="json"
>> interval="60"
>> log.syslog="off"
>> log.file="/var/log/rsyslog-stats.log"
>> severity="7")
>>
>> if $programname startswith 'foo.' then {
>> action(type="omfwd"
>> action.resumeRetryCount="-1"
>> name="spooling"
>> target="analytics"
>> port="514"
>> protocol="tcp"
>> queue.filename="eggforward"
>> queue.spoolDirectory="/var/spool/rsyslog"
>> queue.type="LinkedList")
>> }
>>
>> Thanks for your help
>> Smana
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
>> LIKE THAT.
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
>> LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
