Re: [rsyslog] [RFC] Log-forward destination-cluster support

[chenlin rao]

Is this suggest any progress?

I find my rsyslog server behind LVS output many error logs like
"rsyslogd-2165: netstream session 0x7ff978739ce0 from 172.16.140.13 will be
closed due to error  [v8.11.0 try http://www.rsyslog.com/e/2165 ]". If I
set the omfwd target of clients to only one server, the server wouldn't
output such message.

Don't sure where the problem is. But I remember this email, and think a
cluster support in client side maybe a good choice.

2015-06-05 1:53 GMT+08:00 David Lang :

> On Thu, 4 Jun 2015, Brian Knox wrote:
>
> I've been pondering other discovery support.  So far, I've been wary of
>> introducing additional dependencies.  There's so many discovery services
>> that are en vogue right now (zookeeper, etcd, consul...) and every
>> additional protocol supported is more overhead from a support and
>> maintenance standpoint.
>>
>
> This is exactly the problem.
>
> If we start implementing specific discovery protocols, they will have to
> be maintained for a long time, even when the fad fades and it's not a
> popular thing any longer.
>
> As far as I know, all of these support using name resolution for discovery
> (either DNS or tweaking /etc/hosts) as well as their various proprietary
> protocols, so as long as rsyslog does a name resolution check every time it
> attempts to establish a new connection, these discovery tools could be
> used. DNS propogation across the Internet is slow, but when used within
> your own network you can disable all caching or set the cache to something
> very short. Normal round-robin DNS (listing multiple IPs for one name)
> should result in queries returning different answers even when caching is
> involved.
>
> So, Rsyslog will connect to whatever the service discovery system says is
> correct, and it can be configured to reconnect every X messages (and it
> will reconnect if it gets disconnected due to any error)
>
> This should mean nothing is needed on the initiating (client) side.
>
> On the receiving (server) side, there shouldn't be anything that needs to
> happen either. The service discovery system needs to setup a health check
> of some kind to decide if rsyslog is healthy or not, and if it is, it is
> part of the cluster, if not, it's cut out. This is no different than using
> any other load balancer (it can be "does it accept a TCP connection", or
> "does it accept a message", or "does it recieve a message and output it to
> a specific destination" depending on how paranoid you are)
>
> unless rsyslog is caching name lookups for outbound connections, there
> should be nothing that needs to be changed in rsyslog.
>
> David Lang
>
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] [RFC] Log-forward destination-cluster support

[David Lang]

That sounds as if your LVS server is dropping the connections, possibly because 
they are idle for too long.


What I suggest is that you add something like

logger -t vmststat vmstat 60

to your machines so that they log at least one log per minute (to keep things 
like firewalls and LVS from timing out the connection), and I think you'll find 
it's useful data to have in the system


If you use clusterlabs.org for your failover instead of a pair of LVS servers 
(which use clusterlabs.org for their failover) you will not have the timeout 
problems that you are having with LVS.


I still think trying to bake cluster awareness into the senders is the wrong 
thing to do, the senders cannot possibly coordinate to avoid overload on 
active-active clusters, or to pick the 'same' system when you want 
active-passive clusters.


There are a lot of things that only can be done in rsyslog, clustering is not 
one of them. Rsyslog has better cluster support than nxlog and logstash (I don't 
know about syslog-ng, haven't had to try with it yet :-) it just takes using the 
right tool for the job rather than expecting the loging softwre to do it all.


remember the unix approach, do one thing and do it well, assemble systems from 
tools that take this approach.


Clustering is rather complex topic. A great book on the topic is "In Search of 
Clusters"


David Lang

On Mon, 12 Oct 2015, chenlin rao wrote:


Is this suggest any progress?

I find my rsyslog server behind LVS output many error logs like
"rsyslogd-2165: netstream session 0x7ff978739ce0 from 172.16.140.13 will be
closed due to error  [v8.11.0 try http://www.rsyslog.com/e/2165 ]". If I
set the omfwd target of clients to only one server, the server wouldn't
output such message.

Don't sure where the problem is. But I remember this email, and think a
cluster support in client side maybe a good choice.

2015-06-05 1:53 GMT+08:00 David Lang :


On Thu, 4 Jun 2015, Brian Knox wrote:

I've been pondering other discovery support.  So far, I've been wary of

introducing additional dependencies.  There's so many discovery services
that are en vogue right now (zookeeper, etcd, consul...) and every
additional protocol supported is more overhead from a support and
maintenance standpoint.



This is exactly the problem.

If we start implementing specific discovery protocols, they will have to
be maintained for a long time, even when the fad fades and it's not a
popular thing any longer.

As far as I know, all of these support using name resolution for discovery
(either DNS or tweaking /etc/hosts) as well as their various proprietary
protocols, so as long as rsyslog does a name resolution check every time it
attempts to establish a new connection, these discovery tools could be
used. DNS propogation across the Internet is slow, but when used within
your own network you can disable all caching or set the cache to something
very short. Normal round-robin DNS (listing multiple IPs for one name)
should result in queries returning different answers even when caching is
involved.

So, Rsyslog will connect to whatever the service discovery system says is
correct, and it can be configured to reconnect every X messages (and it
will reconnect if it gets disconnected due to any error)

This should mean nothing is needed on the initiating (client) side.

On the receiving (server) side, there shouldn't be anything that needs to
happen either. The service discovery system needs to setup a health check
of some kind to decide if rsyslog is healthy or not, and if it is, it is
part of the cluster, if not, it's cut out. This is no different than using
any other load balancer (it can be "does it accept a TCP connection", or
"does it accept a message", or "does it recieve a message and output it to
a specific destination" depending on how paranoid you are)

unless rsyslog is caching name lookups for outbound connections, there
should be nothing that needs to be changed in rsyslog.

David Lang

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow 

Re: [rsyslog] [RFC] Log-forward destination-cluster support

[Brian Knox]

David -

I agree that RELP would be the right place for it.  For TCP load balancing
with rsyslog currently, I find using an external load balancer such as
haproxy works nicely.

Brian

On Thu, Jun 4, 2015 at 1:40 PM, David Lang da...@lang.hm wrote:

 If we do decide to do this, it would be better to base the work on relp
 than tcp (if the purpose is reliable delivery under failure conditions)

 The thing is that failover and load balancing can be a rather complex
 problem with many different solutions (different ones are better in
 different conditions). Trying to implement the best options of everything
 inside rsyslog is a lot of work, and I'd prefer the time being spent on
 improving the things that can't be done with exiting tools :-)

 Rsyslog already has better support for load balancing than logstash and
 nxlog (I haven't looked at syslog-ng)

 One question, if an action is configured to go to a name, when it
 reconnects does it do another name lookup? or is it cached?


 On Thu, 4 Jun 2015, singh.janmejay wrote:

  Yes L4 load-balancing will work to significant scale. L7
 load-balancing will do even better in terms of even load, but not sure
 if syslog protocol is widely supported in load-balancers.


 The syslog protocol is not supported by load balancers at L7.

 However, this is still one of the places where existing load balancing
 solutions would do better than your proposed solution. Having each client
 connect randomly would result in more even load balancing only if they are
 all generating the same amount of traffic. Since they aren't, it's going to
 be uneven, and the clients cannot know what the right thing to do is.

 Doing L2 load balancing at the destination, the load balancer can see all
 the traffic and make descisions on it.

  DNS scaling and propagation delay are sometimes not acceptable, but
 BGP anycast is something that'd work at data-center scale with very
 large PODs.


 DNS and BGP failovers within your own network are as fast as you configure
 them to be :-). I'm not even saying BGP anycast, just normal BGP failover
 for when a set of IPs becomes unavailable, route them to a different
 destination.

  This is an alternative to that. It has fewer moving parts (just
 producer and consumer), no LB and it doesn't require the complexity of
 anycast.


 on the other hand, it requires much more complex configuration on every
 client. Every time there is a change on the number of systems in the
 cluster, every single client must be updated, or they will only deliver to
 a subset of the available systems. From a sysadmin point of view, this is a
 horrible thing to maintain. It's possible if you have a centralized config
 management system, but that's a lot more complexity.

  It trades-off engineering complexity of load-balancer and anycast with
 smarter-clients and servers (increasing the complexity of clients and
 servers a little, but also simplifying the deployment topology
 significantly).


 I see this as being a significantly more complex deployment topology :-)

  I think all three are valid approaches and choice of one over the
 other(best fit) will vary across deployments.


 The question I have is if the value of adding this option in rsyslog is
 greater than the features that would be added instead.

 David Lang

 ___
 rsyslog mailing list
 http://lists.adiscon.net/mailman/listinfo/rsyslog
 http://www.rsyslog.com/professional-services/
 What's up with rsyslog? Follow https://twitter.com/rgerhards
 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
 of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
 DON'T LIKE THAT.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] [RFC] Log-forward destination-cluster support

[David Lang]

If we do decide to do this, it would be better to base the work on relp than tcp 
(if the purpose is reliable delivery under failure conditions)


The thing is that failover and load balancing can be a rather complex problem 
with many different solutions (different ones are better in different 
conditions). Trying to implement the best options of everything inside rsyslog 
is a lot of work, and I'd prefer the time being spent on improving the things 
that can't be done with exiting tools :-)


Rsyslog already has better support for load balancing than logstash and nxlog (I 
haven't looked at syslog-ng)


One question, if an action is configured to go to a name, when it reconnects 
does it do another name lookup? or is it cached?



On Thu, 4 Jun 2015, singh.janmejay wrote:


Yes L4 load-balancing will work to significant scale. L7
load-balancing will do even better in terms of even load, but not sure
if syslog protocol is widely supported in load-balancers.


The syslog protocol is not supported by load balancers at L7.

However, this is still one of the places where existing load balancing solutions 
would do better than your proposed solution. Having each client connect randomly 
would result in more even load balancing only if they are all generating the 
same amount of traffic. Since they aren't, it's going to be uneven, and the 
clients cannot know what the right thing to do is.


Doing L2 load balancing at the destination, the load balancer can see all the 
traffic and make descisions on it.



DNS scaling and propagation delay are sometimes not acceptable, but
BGP anycast is something that'd work at data-center scale with very
large PODs.


DNS and BGP failovers within your own network are as fast as you configure them 
to be :-). I'm not even saying BGP anycast, just normal BGP failover for when a 
set of IPs becomes unavailable, route them to a different destination.



This is an alternative to that. It has fewer moving parts (just
producer and consumer), no LB and it doesn't require the complexity of
anycast.


on the other hand, it requires much more complex configuration on every client. 
Every time there is a change on the number of systems in the cluster, every 
single client must be updated, or they will only deliver to a subset of the 
available systems. From a sysadmin point of view, this is a horrible thing to 
maintain. It's possible if you have a centralized config management system, but 
that's a lot more complexity.



It trades-off engineering complexity of load-balancer and anycast with
smarter-clients and servers (increasing the complexity of clients and
servers a little, but also simplifying the deployment topology
significantly).


I see this as being a significantly more complex deployment topology :-)


I think all three are valid approaches and choice of one over the
other(best fit) will vary across deployments.


The question I have is if the value of adding this option in rsyslog is greater 
than the features that would be added instead.


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] [RFC] Log-forward destination-cluster support

[David Lang]

On Thu, 4 Jun 2015, Brian Knox wrote:


I've been pondering other discovery support.  So far, I've been wary of
introducing additional dependencies.  There's so many discovery services
that are en vogue right now (zookeeper, etcd, consul...) and every
additional protocol supported is more overhead from a support and
maintenance standpoint.


This is exactly the problem.

If we start implementing specific discovery protocols, they will have to be 
maintained for a long time, even when the fad fades and it's not a popular thing 
any longer.


As far as I know, all of these support using name resolution for discovery
(either DNS or tweaking /etc/hosts) as well as their various proprietary 
protocols, so as long as rsyslog does a name resolution check every time it 
attempts to establish a new connection, these discovery tools could be used. DNS 
propogation across the Internet is slow, but when used within your own network 
you can disable all caching or set the cache to something very short. Normal 
round-robin DNS (listing multiple IPs for one name) should result in queries 
returning different answers even when caching is involved.


So, Rsyslog will connect to whatever the service discovery system says is 
correct, and it can be configured to reconnect every X messages (and it will 
reconnect if it gets disconnected due to any error)


This should mean nothing is needed on the initiating (client) side.

On the receiving (server) side, there shouldn't be anything that needs to happen 
either. The service discovery system needs to setup a health check of some kind 
to decide if rsyslog is healthy or not, and if it is, it is part of the cluster, 
if not, it's cut out. This is no different than using any other load balancer 
(it can be does it accept a TCP connection, or does it accept a message, or 
does it recieve a message and output it to a specific destination depending on 
how paranoid you are)


unless rsyslog is caching name lookups for outbound connections, there should be 
nothing that needs to be changed in rsyslog.


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] [RFC] Log-forward destination-cluster support

[Brian Knox]

You might want to take a look at the omczmq and imczmq (the new ZeroMQ
input and output plugins I've been working on).  See -
https://github.com/rsyslog/rsyslog/tree/master/contrib/omczmq  and
https://github.com/rsyslog/rsyslog/tree/master/contrib/imczmq

Out of the box they currently support fan out / fan in  and  pub-sub (
note: pub sub does not apply backpressure - it's meant to be lossy in
zeromq). topologies.  I recently started adding support for zbeacon (
http://czmq.zeromq.org/manual:zbeacon ), a UDP based discovery service.
While UDP multicast is not the best solution everywhere, it works for some
cases.

My short to medium term plans going forward with these plugins include:

* Support for non encrypted connections (curvezmq encryption was my first
priority, the plugins don't actually have options for non encrypted
connections yet).
* credit based flow control
* malamute (an embedded broker - see https://github.com/zeromq/malamute )

I've been pondering other discovery support.  So far, I've been wary of
introducing additional dependencies.  There's so many discovery services
that are en vogue right now (zookeeper, etcd, consul...) and every
additional protocol supported is more overhead from a support and
maintenance standpoint.  So at the moment I've been focused solely on ZMTP
( the protocol libzmq implements - http://rfc.zeromq.org/spec:23 )

Brian









On Thu, Jun 4, 2015 at 4:46 AM, singh.janmejay singh.janme...@gmail.com
wrote:

 It won't be a very large change really if we develop it in an external
 library.

 In rsyslog codebase, its a fairly small change, limited to input and
 output modules that we pick. It'll be small parts of plugin code
 (where new connection is established that will call this library
 function conditionally, thats about it).



 On Thu, Jun 4, 2015 at 2:09 PM, Rainer Gerhards
 rgerha...@hq.adiscon.com wrote:
  Sorry if this sounds discouraging: I currently have such a large
  backlog that I can't engage in that effort and I think I am also
  unable to merge any change of this magnitude any time before the
  backlog has become shorter (Q4+ 2015 I guess).
 
  Sorry I have no better answer, but you see yourself what all is going
  on and I really need to make sure I can follow at least the bare
  essentials.
 
  Rainer
 
  2015-06-04 5:53 GMT+02:00 singh.janmejay singh.janme...@gmail.com:
  Yes L4 load-balancing will work to significant scale. L7
  load-balancing will do even better in terms of even load, but not sure
  if syslog protocol is widely supported in load-balancers.
 
  DNS scaling and propagation delay are sometimes not acceptable, but
  BGP anycast is something that'd work at data-center scale with very
  large PODs.
 
  This is an alternative to that. It has fewer moving parts (just
  producer and consumer), no LB and it doesn't require the complexity of
  anycast.
 
  It trades-off engineering complexity of load-balancer and anycast with
  smarter-clients and servers (increasing the complexity of clients and
  servers a little, but also simplifying the deployment topology
  significantly).
 
  I think all three are valid approaches and choice of one over the
  other(best fit) will vary across deployments.
 
 
  On Thu, Jun 4, 2015 at 8:45 AM, David Lang da...@lang.hm wrote:
  I don't see the advantage of adding all this complexity as opposed to
 using
  existing load balancing approaches. With existing tools we can deliver
 the
  log stream to a cluster of systems, and deal with them failing. Yes,
 the
  easy approaches to doing this are limited to the throughput of a single
  wire, but since that single wire is commonly 10Gb/sec (and easily
 40Gb/sec)
  with off-the-shelf technology, and the fact that the log stream can be
  compressed, this isn't likely to be an issue for much of anyone below
 Google
  scale.
 
  There is a lot of advantages to keeping the failover logic and config
  contained to as small an area of the network and as few devices as
 possible.
  The systems accepting the ogs _must- participate in the process
 (responding
  to health checks if nothing else), it only takes a couple other boxes
 (if
  any) to perform TCP load balancing. And having everything local
 increases
  the accuracy of the detection and speed of recovery.
 
  If you want to deal with larger failures (datacenter scale), then
 existing
  DNS/BGP failover tools can come into play.
 
  What advantage do we gain by pushing the configuration and failover
 logic to
  the senders?
 
  David Lang
 
 
  On Thu, 4 Jun 2015, singh.janmejay wrote:
 
  Hi,
 
  This is proposal towards first-class support for notion of a 'cluster'
  as a log-forwarding destination. It talks about a
  technology-independent service-discovery-support implementation.
 
  Scenario / Context:
  
  Say an environment is supposed to relay all logs to a logical
  destination for aggregation/archival purpose. Such a setup at large
  scale would have a several log-producers 

Re: [rsyslog] [RFC] Log-forward destination-cluster support

[Rainer Gerhards]

Sorry if this sounds discouraging: I currently have such a large
backlog that I can't engage in that effort and I think I am also
unable to merge any change of this magnitude any time before the
backlog has become shorter (Q4+ 2015 I guess).

Sorry I have no better answer, but you see yourself what all is going
on and I really need to make sure I can follow at least the bare
essentials.

Rainer

2015-06-04 5:53 GMT+02:00 singh.janmejay singh.janme...@gmail.com:
 Yes L4 load-balancing will work to significant scale. L7
 load-balancing will do even better in terms of even load, but not sure
 if syslog protocol is widely supported in load-balancers.

 DNS scaling and propagation delay are sometimes not acceptable, but
 BGP anycast is something that'd work at data-center scale with very
 large PODs.

 This is an alternative to that. It has fewer moving parts (just
 producer and consumer), no LB and it doesn't require the complexity of
 anycast.

 It trades-off engineering complexity of load-balancer and anycast with
 smarter-clients and servers (increasing the complexity of clients and
 servers a little, but also simplifying the deployment topology
 significantly).

 I think all three are valid approaches and choice of one over the
 other(best fit) will vary across deployments.


 On Thu, Jun 4, 2015 at 8:45 AM, David Lang da...@lang.hm wrote:
 I don't see the advantage of adding all this complexity as opposed to using
 existing load balancing approaches. With existing tools we can deliver the
 log stream to a cluster of systems, and deal with them failing. Yes, the
 easy approaches to doing this are limited to the throughput of a single
 wire, but since that single wire is commonly 10Gb/sec (and easily 40Gb/sec)
 with off-the-shelf technology, and the fact that the log stream can be
 compressed, this isn't likely to be an issue for much of anyone below Google
 scale.

 There is a lot of advantages to keeping the failover logic and config
 contained to as small an area of the network and as few devices as possible.
 The systems accepting the ogs _must- participate in the process (responding
 to health checks if nothing else), it only takes a couple other boxes (if
 any) to perform TCP load balancing. And having everything local increases
 the accuracy of the detection and speed of recovery.

 If you want to deal with larger failures (datacenter scale), then existing
 DNS/BGP failover tools can come into play.

 What advantage do we gain by pushing the configuration and failover logic to
 the senders?

 David Lang


 On Thu, 4 Jun 2015, singh.janmejay wrote:

 Hi,

 This is proposal towards first-class support for notion of a 'cluster'
 as a log-forwarding destination. It talks about a
 technology-independent service-discovery-support implementation.

 Scenario / Context:
 
 Say an environment is supposed to relay all logs to a logical
 destination for aggregation/archival purpose. Such a setup at large
 scale would have a several log-producers and several log-receivers (or
 listeners).

 Because of several machines being involved, probability of some
 listener nodes failing is significant, and log-producers should
 ideally be somehow aware of current log-receiving-cluster definition
 (that is, nodes that are healthy and receiving logs).

 This proposal talks about first-class definition and usage of
 log-receiver-cluster.

 Configuration concepts:
 -

 action(...): gets an additional parameter called cluster, which
 follows URL like conversion (scheme is used to indicate technology
 used for service-discovery).

 input(...): gets an additional parameter called cluster, same as action

 The URL must resolve to currently-available host+port pairs, and may
 support notifications, polling or both (depends on underlying
 service-discovery technology).

 The resolution of URLs and representation of host+port pairs in the
 service-endpoint is left to service-discovery technology.

 Configuration examples:
 --

 action(type=omfwd cluster=zk://10.20.30.40,10.20.30.41/foo_cluster
 template=foo-fwd protocol=tcp tcp_framing=octet-counted
 ResendLastMSGOnReconnect=on)

 input(type=imptcp port=10514 name=foo ruleset=foo.recv
 cluster=zk://10.20.30.40,10.20.30.41/foo_cluster)

 The parameter cluster in both cases points to a url
 zk://10.20.30.40,10.20.30.41/foo_cluster which does:

 [behaviour common to both input and action]
 - scheme 'zk' resolves to use of zookeeper based
 service-discovery (picks one of the multiple supported
 service-discovery technologies)
 - reach the zookeeper at IP address: 10.20.30.40 or 10.20.30.41
 - look for znode (zookeeper specific): /foo_cluster


 [action specific behaviour]
 - enumerate ephemeral nodes under the said znode
 - pick one of the child-nodes at random
 - expect it to be follow naming convension host:port
 - break host and port apart and use it as destination for 

Re: [rsyslog] [RFC] Log-forward destination-cluster support

[singh.janmejay]

It won't be a very large change really if we develop it in an external library.

In rsyslog codebase, its a fairly small change, limited to input and
output modules that we pick. It'll be small parts of plugin code
(where new connection is established that will call this library
function conditionally, thats about it).



On Thu, Jun 4, 2015 at 2:09 PM, Rainer Gerhards
rgerha...@hq.adiscon.com wrote:
 Sorry if this sounds discouraging: I currently have such a large
 backlog that I can't engage in that effort and I think I am also
 unable to merge any change of this magnitude any time before the
 backlog has become shorter (Q4+ 2015 I guess).

 Sorry I have no better answer, but you see yourself what all is going
 on and I really need to make sure I can follow at least the bare
 essentials.

 Rainer

 2015-06-04 5:53 GMT+02:00 singh.janmejay singh.janme...@gmail.com:
 Yes L4 load-balancing will work to significant scale. L7
 load-balancing will do even better in terms of even load, but not sure
 if syslog protocol is widely supported in load-balancers.

 DNS scaling and propagation delay are sometimes not acceptable, but
 BGP anycast is something that'd work at data-center scale with very
 large PODs.

 This is an alternative to that. It has fewer moving parts (just
 producer and consumer), no LB and it doesn't require the complexity of
 anycast.

 It trades-off engineering complexity of load-balancer and anycast with
 smarter-clients and servers (increasing the complexity of clients and
 servers a little, but also simplifying the deployment topology
 significantly).

 I think all three are valid approaches and choice of one over the
 other(best fit) will vary across deployments.


 On Thu, Jun 4, 2015 at 8:45 AM, David Lang da...@lang.hm wrote:
 I don't see the advantage of adding all this complexity as opposed to using
 existing load balancing approaches. With existing tools we can deliver the
 log stream to a cluster of systems, and deal with them failing. Yes, the
 easy approaches to doing this are limited to the throughput of a single
 wire, but since that single wire is commonly 10Gb/sec (and easily 40Gb/sec)
 with off-the-shelf technology, and the fact that the log stream can be
 compressed, this isn't likely to be an issue for much of anyone below Google
 scale.

 There is a lot of advantages to keeping the failover logic and config
 contained to as small an area of the network and as few devices as possible.
 The systems accepting the ogs _must- participate in the process (responding
 to health checks if nothing else), it only takes a couple other boxes (if
 any) to perform TCP load balancing. And having everything local increases
 the accuracy of the detection and speed of recovery.

 If you want to deal with larger failures (datacenter scale), then existing
 DNS/BGP failover tools can come into play.

 What advantage do we gain by pushing the configuration and failover logic to
 the senders?

 David Lang


 On Thu, 4 Jun 2015, singh.janmejay wrote:

 Hi,

 This is proposal towards first-class support for notion of a 'cluster'
 as a log-forwarding destination. It talks about a
 technology-independent service-discovery-support implementation.

 Scenario / Context:
 
 Say an environment is supposed to relay all logs to a logical
 destination for aggregation/archival purpose. Such a setup at large
 scale would have a several log-producers and several log-receivers (or
 listeners).

 Because of several machines being involved, probability of some
 listener nodes failing is significant, and log-producers should
 ideally be somehow aware of current log-receiving-cluster definition
 (that is, nodes that are healthy and receiving logs).

 This proposal talks about first-class definition and usage of
 log-receiver-cluster.

 Configuration concepts:
 -

 action(...): gets an additional parameter called cluster, which
 follows URL like conversion (scheme is used to indicate technology
 used for service-discovery).

 input(...): gets an additional parameter called cluster, same as action

 The URL must resolve to currently-available host+port pairs, and may
 support notifications, polling or both (depends on underlying
 service-discovery technology).

 The resolution of URLs and representation of host+port pairs in the
 service-endpoint is left to service-discovery technology.

 Configuration examples:
 --

 action(type=omfwd cluster=zk://10.20.30.40,10.20.30.41/foo_cluster
 template=foo-fwd protocol=tcp tcp_framing=octet-counted
 ResendLastMSGOnReconnect=on)

 input(type=imptcp port=10514 name=foo ruleset=foo.recv
 cluster=zk://10.20.30.40,10.20.30.41/foo_cluster)

 The parameter cluster in both cases points to a url
 zk://10.20.30.40,10.20.30.41/foo_cluster which does:

 [behaviour common to both input and action]
 - scheme 'zk' resolves to use of zookeeper based
 service-discovery (picks one of the multiple supported
 

Re: [rsyslog] [RFC] Log-forward destination-cluster support

[David Lang]

I don't see the advantage of adding all this complexity as opposed to using 
existing load balancing approaches. With existing tools we can deliver the log 
stream to a cluster of systems, and deal with them failing. Yes, the easy 
approaches to doing this are limited to the throughput of a single wire, but 
since that single wire is commonly 10Gb/sec (and easily 40Gb/sec) with 
off-the-shelf technology, and the fact that the log stream can be compressed, 
this isn't likely to be an issue for much of anyone below Google scale.


There is a lot of advantages to keeping the failover logic and config contained 
to as small an area of the network and as few devices as possible. The systems 
accepting the ogs _must- participate in the process (responding to health checks 
if nothing else), it only takes a couple other boxes (if any) to perform TCP 
load balancing. And having everything local increases the accuracy of the 
detection and speed of recovery.


If you want to deal with larger failures (datacenter scale), then existing 
DNS/BGP failover tools can come into play.


What advantage do we gain by pushing the configuration and failover logic to the 
senders?


David Lang

On Thu, 4 Jun 2015, singh.janmejay wrote:


Hi,

This is proposal towards first-class support for notion of a 'cluster'
as a log-forwarding destination. It talks about a
technology-independent service-discovery-support implementation.

Scenario / Context:

Say an environment is supposed to relay all logs to a logical
destination for aggregation/archival purpose. Such a setup at large
scale would have a several log-producers and several log-receivers (or
listeners).

Because of several machines being involved, probability of some
listener nodes failing is significant, and log-producers should
ideally be somehow aware of current log-receiving-cluster definition
(that is, nodes that are healthy and receiving logs).

This proposal talks about first-class definition and usage of
log-receiver-cluster.

Configuration concepts:
-

action(...): gets an additional parameter called cluster, which
follows URL like conversion (scheme is used to indicate technology
used for service-discovery).

input(...): gets an additional parameter called cluster, same as action

The URL must resolve to currently-available host+port pairs, and may
support notifications, polling or both (depends on underlying
service-discovery technology).

The resolution of URLs and representation of host+port pairs in the
service-endpoint is left to service-discovery technology.

Configuration examples:
--

action(type=omfwd cluster=zk://10.20.30.40,10.20.30.41/foo_cluster
template=foo-fwd protocol=tcp tcp_framing=octet-counted
ResendLastMSGOnReconnect=on)

input(type=imptcp port=10514 name=foo ruleset=foo.recv
cluster=zk://10.20.30.40,10.20.30.41/foo_cluster)

The parameter cluster in both cases points to a url
zk://10.20.30.40,10.20.30.41/foo_cluster which does:

[behaviour common to both input and action]
- scheme 'zk' resolves to use of zookeeper based
service-discovery (picks one of the multiple supported
service-discovery technologies)
- reach the zookeeper at IP address: 10.20.30.40 or 10.20.30.41
- look for znode (zookeeper specific): /foo_cluster


[action specific behaviour]
- enumerate ephemeral nodes under the said znode
- pick one of the child-nodes at random
- expect it to be follow naming convension host:port
- break host and port apart and use it as destination for forwarding logs


[input specific behaviour]
- enumerate IP addresses that this input is listening to
- create ephemeral child-znodes under the foo_cluster znode with
name IP:port (follow this convention, enforced by service discovery
technology specific implementation)
- keep the session live as long as input is live (which means,
kill the session if queues are filled up, or the heartbeat-mechanism
will automatically kill it if rsyslog process dies, or host freezes up
or fails, or connecting switch fails, etc).

Service-discovery technology independence:


URL-like naming convention allows for scheme to indicate
service-discovery technology. Cluster-url for simple polling based
discovery technology implemented over http would look like this:

http://10.20.30.40/foo_cluster

A log-cabin based impl would have something similar to:

logcabin://10.20.30.40,10.20.30.40/foo_cluster

Implementation sketch:


Service-discovery support can be implemented within rsyslog(runtime),
or as an independent library under rsyslog umbrella (I think I prefer
independent library).

Any input or output modules that choose to support it basically link
with the library and do one of the following:

- as an action:
when doAction fails, it picks a random-endpoint from a collection
of currently-available-endpoints provided 

[rsyslog] [RFC] Log-forward destination-cluster support

[singh.janmejay]

Hi,

This is proposal towards first-class support for notion of a 'cluster'
as a log-forwarding destination. It talks about a
technology-independent service-discovery-support implementation.

Scenario / Context:

Say an environment is supposed to relay all logs to a logical
destination for aggregation/archival purpose. Such a setup at large
scale would have a several log-producers and several log-receivers (or
listeners).

Because of several machines being involved, probability of some
listener nodes failing is significant, and log-producers should
ideally be somehow aware of current log-receiving-cluster definition
(that is, nodes that are healthy and receiving logs).

This proposal talks about first-class definition and usage of
log-receiver-cluster.

Configuration concepts:
-

action(...): gets an additional parameter called cluster, which
follows URL like conversion (scheme is used to indicate technology
used for service-discovery).

input(...): gets an additional parameter called cluster, same as action

The URL must resolve to currently-available host+port pairs, and may
support notifications, polling or both (depends on underlying
service-discovery technology).

The resolution of URLs and representation of host+port pairs in the
service-endpoint is left to service-discovery technology.

Configuration examples:
--

action(type=omfwd cluster=zk://10.20.30.40,10.20.30.41/foo_cluster
template=foo-fwd protocol=tcp tcp_framing=octet-counted
ResendLastMSGOnReconnect=on)

input(type=imptcp port=10514 name=foo ruleset=foo.recv
cluster=zk://10.20.30.40,10.20.30.41/foo_cluster)

The parameter cluster in both cases points to a url
zk://10.20.30.40,10.20.30.41/foo_cluster which does:

[behaviour common to both input and action]
 - scheme 'zk' resolves to use of zookeeper based
service-discovery (picks one of the multiple supported
service-discovery technologies)
 - reach the zookeeper at IP address: 10.20.30.40 or 10.20.30.41
 - look for znode (zookeeper specific): /foo_cluster


[action specific behaviour]
 - enumerate ephemeral nodes under the said znode
 - pick one of the child-nodes at random
 - expect it to be follow naming convension host:port
 - break host and port apart and use it as destination for forwarding logs


[input specific behaviour]
 - enumerate IP addresses that this input is listening to
 - create ephemeral child-znodes under the foo_cluster znode with
name IP:port (follow this convention, enforced by service discovery
technology specific implementation)
 - keep the session live as long as input is live (which means,
kill the session if queues are filled up, or the heartbeat-mechanism
will automatically kill it if rsyslog process dies, or host freezes up
or fails, or connecting switch fails, etc).

Service-discovery technology independence:


URL-like naming convention allows for scheme to indicate
service-discovery technology. Cluster-url for simple polling based
discovery technology implemented over http would look like this:

http://10.20.30.40/foo_cluster

A log-cabin based impl would have something similar to:

logcabin://10.20.30.40,10.20.30.40/foo_cluster

Implementation sketch:


Service-discovery support can be implemented within rsyslog(runtime),
or as an independent library under rsyslog umbrella (I think I prefer
independent library).

Any input or output modules that choose to support it basically link
with the library and do one of the following:

- as an action:
 when doAction fails, it picks a random-endpoint from a collection
of currently-available-endpoints provided by service discovery (random
picking from the collection is implemented inside the library, because
then it allows us later to make use of other parameters (such as
current load, which will be better for load-balancing)).

- as an input:
 if listen is successful and input is ready to take traffic, it
enumerates all IP addresses it is listening on, and registers with
service discovery technology those end-points (which can then be
discovered by senders).

This can also be used at source or sink where producer or consumer is
aware of the service-discovery technology specific protocol and
rsyslog is only at one end of the pipe (as long as the same protocol
is followed, which will be easy considering the implementation is
available as a library).

Host and port as required by to-be-supported input/output module will
become optional. Either host+port or cluster parameter must be
specified, specifying both will generate a warning, and it'll discard
cluster in favour of host+port (based on specificity).

The support for this can be built incrementally for each input/output
module as desired, I have omfwd and imptcp in mind at this time.

Thoughts?

-- 
Regards,
Janmejay
http://codehunk.wordpress.com

Re: [rsyslog] [RFC] Log-forward destination-cluster support

[singh.janmejay]

Yes L4 load-balancing will work to significant scale. L7
load-balancing will do even better in terms of even load, but not sure
if syslog protocol is widely supported in load-balancers.

DNS scaling and propagation delay are sometimes not acceptable, but
BGP anycast is something that'd work at data-center scale with very
large PODs.

This is an alternative to that. It has fewer moving parts (just
producer and consumer), no LB and it doesn't require the complexity of
anycast.

It trades-off engineering complexity of load-balancer and anycast with
smarter-clients and servers (increasing the complexity of clients and
servers a little, but also simplifying the deployment topology
significantly).

I think all three are valid approaches and choice of one over the
other(best fit) will vary across deployments.


On Thu, Jun 4, 2015 at 8:45 AM, David Lang da...@lang.hm wrote:
 I don't see the advantage of adding all this complexity as opposed to using
 existing load balancing approaches. With existing tools we can deliver the
 log stream to a cluster of systems, and deal with them failing. Yes, the
 easy approaches to doing this are limited to the throughput of a single
 wire, but since that single wire is commonly 10Gb/sec (and easily 40Gb/sec)
 with off-the-shelf technology, and the fact that the log stream can be
 compressed, this isn't likely to be an issue for much of anyone below Google
 scale.

 There is a lot of advantages to keeping the failover logic and config
 contained to as small an area of the network and as few devices as possible.
 The systems accepting the ogs _must- participate in the process (responding
 to health checks if nothing else), it only takes a couple other boxes (if
 any) to perform TCP load balancing. And having everything local increases
 the accuracy of the detection and speed of recovery.

 If you want to deal with larger failures (datacenter scale), then existing
 DNS/BGP failover tools can come into play.

 What advantage do we gain by pushing the configuration and failover logic to
 the senders?

 David Lang


 On Thu, 4 Jun 2015, singh.janmejay wrote:

 Hi,

 This is proposal towards first-class support for notion of a 'cluster'
 as a log-forwarding destination. It talks about a
 technology-independent service-discovery-support implementation.

 Scenario / Context:
 
 Say an environment is supposed to relay all logs to a logical
 destination for aggregation/archival purpose. Such a setup at large
 scale would have a several log-producers and several log-receivers (or
 listeners).

 Because of several machines being involved, probability of some
 listener nodes failing is significant, and log-producers should
 ideally be somehow aware of current log-receiving-cluster definition
 (that is, nodes that are healthy and receiving logs).

 This proposal talks about first-class definition and usage of
 log-receiver-cluster.

 Configuration concepts:
 -

 action(...): gets an additional parameter called cluster, which
 follows URL like conversion (scheme is used to indicate technology
 used for service-discovery).

 input(...): gets an additional parameter called cluster, same as action

 The URL must resolve to currently-available host+port pairs, and may
 support notifications, polling or both (depends on underlying
 service-discovery technology).

 The resolution of URLs and representation of host+port pairs in the
 service-endpoint is left to service-discovery technology.

 Configuration examples:
 --

 action(type=omfwd cluster=zk://10.20.30.40,10.20.30.41/foo_cluster
 template=foo-fwd protocol=tcp tcp_framing=octet-counted
 ResendLastMSGOnReconnect=on)

 input(type=imptcp port=10514 name=foo ruleset=foo.recv
 cluster=zk://10.20.30.40,10.20.30.41/foo_cluster)

 The parameter cluster in both cases points to a url
 zk://10.20.30.40,10.20.30.41/foo_cluster which does:

 [behaviour common to both input and action]
 - scheme 'zk' resolves to use of zookeeper based
 service-discovery (picks one of the multiple supported
 service-discovery technologies)
 - reach the zookeeper at IP address: 10.20.30.40 or 10.20.30.41
 - look for znode (zookeeper specific): /foo_cluster


 [action specific behaviour]
 - enumerate ephemeral nodes under the said znode
 - pick one of the child-nodes at random
 - expect it to be follow naming convension host:port
 - break host and port apart and use it as destination for forwarding
 logs


 [input specific behaviour]
 - enumerate IP addresses that this input is listening to
 - create ephemeral child-znodes under the foo_cluster znode with
 name IP:port (follow this convention, enforced by service discovery
 technology specific implementation)
 - keep the session live as long as input is live (which means,
 kill the session if queues are filled up, or the heartbeat-mechanism
 will automatically kill it if rsyslog process dies, or host