On 9/28/2017 6:37 AM, Joan via rsyslog wrote:
I am using rsylog to read (imfile) some logfiles and forward them to
graylog in syslog format RFC5424.
I would like to add an attribute with the filename, I explored doing it by
modifiing the template (such as this
<http://kb.monitorware.com/identify-source-filename-remote-logging-t12694.html>).
But it will just add the information into the begin of the message,
breaking some extractors I have in place.
I though after reading about the RFC5424 that another option would be to
add this data into the structured data fields (such as origin), but I
couldn't find any example on how to add this information. Could you point
me to some docs?
Regards,
Joan
Your post to the list reminded me of a task I've yet to start on, but
hope to soon.
I too am currently forwarding from rsyslog to graylog using the
RSYSLOG_SyslogProtocol23Format template and I also hope to extend the
data I'm sending into Graylog to include metadata details like source
filenames or inputs where a message is received.
Because I've recently been learning to use JSON payloads for shipping
extra information (In a structured format), I looked up whether Graylog
accepted log data in a JSON format. Turns out that not only does it
support it, but it appears to accept it in a way that you can easily
have Graylog create additional fields for you from supplied data.
I found a guide[1] for constructing an appropriate GELF template and
plan to use that provided template as a starting point:
template(name="gelf" type="list") {
constant(value="{\"version\":\"1.1\",")
constant(value="\"host\":\"")
property(name="hostname")
constant(value="\",\"short_message\":\"")
property(name="msg" format="json")
constant(value="\",\"timestamp\":\"")
property(name="timegenerated" dateformat="unixtimestamp")
constant(value="\",\"level\":\"")
property(name="syslogseverity")
constant(value="\"}")
}
I say starting point because the Graylog documentation[2] covers
extending the GELF payload to include additional data that Graylog will
automatically treat as an additional field. That I believe refers to the
metadata we wish to push into Graylog.
If this approach interests you, you may wish to read over the resources
I've linked to below. Because I will be switching from the
RSYSLOG_SyslogProtocol23Format template to a GELF TCP Graylog input,
I've recorded in my TODO entry that I'll need to pay attention to these
points:
* GELF TCP only supports uncompressed and non-chunked payloads. Each
message needs to be delimited with a null byte (\0) when sent in the
same TCP connection[2][3].
* Graylog is able to accept and parse RFC 5424 and RFC 3164 compliant
syslog messages and supports TCP transport with both the octet counting
or termination character methods[2][3].
In particular, because I've not changed the default value, I'm still
using the "traditional" setting for the TCP_Framing parameter in my
forwarding action from rsyslog to Graylog. According to what I've read
from the sources below (this[4] one specifically mentions it), I should
probably update that forwarding action to use the the octet-based TCP
Framing option. This appears to be especially important with multi-line
messages.
Not sure how that would apply if the message is entirely composed of a
JSON structure vs a RFC 5424 or RFC 3164 compliant message.
[1] http://www.rsyslog.com/doc/v8-stable/tutorials/gelf_forwarding.html
[2] http://docs.graylog.org/en/2.3/pages/gelf.html#gelf-via-tcp
[3] http://www.rsyslog.com/doc/v8-stable/configuration/modules/omfwd.html
[4]
https://selivan.github.io/2017/02/07/rsyslog-log-forward-save-filename-handle-multi-line-failover.html
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
