On Wed, 23 Jan 2013, Phil Pennock wrote:
So there are two protocols calling themselves "syslog"; RFC 3164 "The
BSD syslog Protocol" and RFC 5424 "The Syslog Protocol" which adds
[idAnd structured=data] after the header, in each message.
Then there's RFC 6587 "Transmission of Syslog Messages over TCP" which
the IETF declared historic, does not support TLS, etc, but basically
describes BSD syslog over TCP, either with newlines as separators or
with octet counting.
My loose understanding, based on no scientific data whatsoever, is that
almost everyone is using BSD syslog, perhaps over TCP with
newline-termination on messages per 6587. I'd like to confirm/deny this
for $employer, to figure out which should be used by default for
something which is directly emitting syslog packets (for good reason).
I see that rsyslog defaults to BSD but also supports the IETF variant.
syslog-ng added support for the IETF stuff when they added TLS support.
What are enterprises (or other large organisations that care about
logging) actually using?
Is anybody using syslog-over-TLS (RFC 5425)? Signed syslog (RFC 5848)?
If using TCP, which syslog variant? "Whatever the default of the
product is" or choosing something specific? And if choosing one in
particular, why?
The nice thing about standards is that there are so many to choose from.
Nowhere is this more true than when dealing with syslog.
The modern syslog daemons (rsyslog, syslog-ng, nxlog, etc) will all support all
of the above, plus other protocols (RELP, Reliable Event Logging Protocol for
example, which does app layer acknowlegements needed to prevent log loss that
can still happen with tcp logging)
There are also several message formats available that you didn't list, including
multiple ones that are trying to standardize the logs (CEE, CEF, Lumberjack,
etc)
You are also going to have to deal with devices that violate every standard
there is and generate log messages that need to be fixed up before they can be
dealt with (and not just from no-name vendors, Cisco is among the offenders)
I just presented a paper a LISA on what we did to build a logging system speced
to handle 100K lines of logs/sec. In testing we ran it up to 380K 250 byte
logs/sec (pretty close to gig-E wire speed)
https://www.usenix.org/conference/lisa12/building-100k-logsec-logging-infrastructure
We are in the middle of discussions for how to build our next generation of
this, expanding from the one division of the company to cover the entire
company.
In general terms, here is what I suggest.
Have redundant log relay systems close to the systems generating the logs, get
the logs into these systems by whatever method is easiest for the system
generating the log (frequently this is plain old BSD syslog via UDP)
These first-tier relay systems do any fixing up of the logs that is needed,
and add metadata to the log messages as needed.
I suggest using the JSON structured log format from these first tier servers. It
gives you a way to add metadata and keep it separate from the log contents.
When I am talking about metadata, I'm thinking of things like what division owns
the system, what the actual IP is of the sending box (as opposed to it's
hostname, or what hostname it puts in the logs), and other information that's
much easier to add early on.
These first-tier relay boxes can then send the logs to a central set of
core relay boxes. This is the point where you break out the more sophisticated
network-layer protocols. You also now worry about queuing logs in the case where
you loose your communications to the core relay boxes.
The core relay boxes then deal with distrubuting the logs to the farms of boxes
that you use to analyse them.
You then can have different, specialized farms of boxes to do the different
types of things that you want done with the logs.
As for the software, I like rsyslog, but reasonable people can disagree on the
best choice for this. I don't think you can go too far wrong with any choice,
they all interoperate pretty well, so you can switch from one to another without
having to change your architecture.
I can go on for a lot longer, either here or in private if you want to talk
about your specifics.
David Lang
_______________________________________________
Discuss mailing list
[email protected]
https://lists.lopsa.org/cgi-bin/mailman/listinfo/discuss
This list provided by the League of Professional System Administrators
http://lopsa.org/
