Re: [rsyslog] Add the file name to syslog data

[David Lang]

Even using the RFC5424 format, I would just use JSON in the message body, the 
structured data idea is something that pretty much nothing uses.


David Lang

On Mon, 9 Oct 2017, Joan via rsyslog wrote:


Date: Mon, 9 Oct 2017 10:53:09 +0200
From: Joan via rsyslog <rsyslog@lists.adiscon.com>
To: rsyslog-users <rsyslog@lists.adiscon.com>
Cc: Joan <aseq...@gmail.com>
Subject: Re: [rsyslog] Add the file name to syslog data

In my case I switched to rfc5424 precisely for the subsecond timestamps,
when agregating data from a lot of places, the messages would get unordered
for some reason, adding the microsecond fixed all that.

Digging in the available choices I amb thinking about two different options:

1) I stumbled into this this
<http://www.rsyslog.com/how-to-add-a-hmac-to-rfc5424-structured-data-messages/>,
the module mmrfc5424addhmac allows adding to the structured data some
information (in this case the hmac value), I looked around the code to
check if there was a generic approach to this, but couldn't find anything.
Is there anything that I have missed? BTW with the mentioned module the SD
can be modified like this:

action(type="mmrfc5424addhmac" key="yourenterprisekey"
hashFunction="sha256" sd_id="id@32473")


2) Another way of doing it would be to compose the structured data by
hand (removing the %STRUCTURED-DATA% currently empty) and replacing it
by a crafted field

$template Protocol23log4j,"<%$.pri%>1 %TIMESTAMP:::date-rfc3339%
%HOSTNAME% %APP-NAME% %PROCID% %MSGID% %STRUCTURED-DATA% %msg%\n"
$template Protocol23log4j,"<%$.pri%>1 %TIMESTAMP:::date-rfc3339%
%HOSTNAME% %APP-NAME% %PROCID% %MSGID% [exampleSDID@32473
file="%$!metadata!filename%"] %msg%\n"

Being the 1) not available if there is no such module, would be the 2)
a valid way of creating this data?



2017-10-07 0:03 GMT+02:00 David Lang <da...@lang.hm>:


On Fri, 6 Oct 2017, Dave Cottlehuber wrote:

at the final destination, I have all that data available and can either

use it, or create a template that just writes out a RFC3164 style message
with the original message content.



Is there any reason why you prefer RFC3164 vs the later RFC5424
http://datatracker.ietf.org/doc/rfc5424 ?



rfc5424 is a failure in many ways and has largely been ignored. Most
things that emit syslogs do so via the older standard, virtually nothing
supports the 'structured data' that it provides (which is why I do json in
the body of the message, everything understands that). most log parsing
tools don't know what to do with the newer format.

and the older format is just more compact (separate fields for app, pid,
and msgid vs the syslogtag app[pid]:)

the one real advantage the new format has is the sub-second timestamp and
the timezone in the timestamp. If you have all your systems running on the
same timezone, then the latter doesn't help, and I really have never had to
worry about sub-second resolution when dealing with logs. It's either the
sequence of the logs or to the nearest minute is good enough.

your milage may vary, there's nothing wrong with using the newer format.

David Lang

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Add the file name to syslog data

[Joan via rsyslog]

In my case I switched to rfc5424 precisely for the subsecond timestamps,
when agregating data from a lot of places, the messages would get unordered
for some reason, adding the microsecond fixed all that.

Digging in the available choices I amb thinking about two different options:

1) I stumbled into this this
,
the module mmrfc5424addhmac allows adding to the structured data some
information (in this case the hmac value), I looked around the code to
check if there was a generic approach to this, but couldn't find anything.
Is there anything that I have missed? BTW with the mentioned module the SD
can be modified like this:

action(type="mmrfc5424addhmac" key="yourenterprisekey"
hashFunction="sha256" sd_id="id@32473")


2) Another way of doing it would be to compose the structured data by
hand (removing the %STRUCTURED-DATA% currently empty) and replacing it
by a crafted field

$template Protocol23log4j,"<%$.pri%>1 %TIMESTAMP:::date-rfc3339%
%HOSTNAME% %APP-NAME% %PROCID% %MSGID% %STRUCTURED-DATA% %msg%\n"
$template Protocol23log4j,"<%$.pri%>1 %TIMESTAMP:::date-rfc3339%
%HOSTNAME% %APP-NAME% %PROCID% %MSGID% [exampleSDID@32473
file="%$!metadata!filename%"] %msg%\n"

Being the 1) not available if there is no such module, would be the 2)
a valid way of creating this data?



2017-10-07 0:03 GMT+02:00 David Lang :

> On Fri, 6 Oct 2017, Dave Cottlehuber wrote:
>
> at the final destination, I have all that data available and can either
>>> use it, or create a template that just writes out a RFC3164 style message
>>> with the original message content.
>>>
>>
>> Is there any reason why you prefer RFC3164 vs the later RFC5424
>> http://datatracker.ietf.org/doc/rfc5424 ?
>>
>
> rfc5424 is a failure in many ways and has largely been ignored. Most
> things that emit syslogs do so via the older standard, virtually nothing
> supports the 'structured data' that it provides (which is why I do json in
> the body of the message, everything understands that). most log parsing
> tools don't know what to do with the newer format.
>
> and the older format is just more compact (separate fields for app, pid,
> and msgid vs the syslogtag app[pid]:)
>
> the one real advantage the new format has is the sub-second timestamp and
> the timezone in the timestamp. If you have all your systems running on the
> same timezone, then the latter doesn't help, and I really have never had to
> worry about sub-second resolution when dealing with logs. It's either the
> sequence of the logs or to the nearest minute is good enough.
>
> your milage may vary, there's nothing wrong with using the newer format.
>
> David Lang
>
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Add the file name to syslog data

[David Lang]

On Fri, 6 Oct 2017, Dave Cottlehuber wrote:

at the final destination, I have all that data available and can either use 
it, or create a template that just writes out a RFC3164 style message with 
the original message content.


Is there any reason why you prefer RFC3164 vs the later RFC5424
http://datatracker.ietf.org/doc/rfc5424 ?


rfc5424 is a failure in many ways and has largely been ignored. Most things that 
emit syslogs do so via the older standard, virtually nothing supports the 
'structured data' that it provides (which is why I do json in the body of the 
message, everything understands that). most log parsing tools don't know what to 
do with the newer format.


and the older format is just more compact (separate fields for app, pid, and 
msgid vs the syslogtag app[pid]:)


the one real advantage the new format has is the sub-second timestamp and the 
timezone in the timestamp. If you have all your systems running on the same 
timezone, then the latter doesn't help, and I really have never had to worry 
about sub-second resolution when dealing with logs. It's either the sequence of 
the logs or to the nearest minute is good enough.


your milage may vary, there's nothing wrong with using the newer format.

David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Add the file name to syslog data

[Dave Cottlehuber]

On Sun, 1 Oct 2017, at 11:50, David Lang wrote:
> any time you have a question like this, first log locally with the format 
> RSYSLOG_DebugFormat so that you can see exactly what data you have where.
> 
> In this case, you will need to enable metadata in your imfile config,
> this will 
> add the filename (and other information), but it doesn't do this by
> changing the 
> message itself, it does this by creating file (under $!)
> 
> so you will need to look at the debug output and then decide how you are
> going 
> to format the output message so that it contains the data.
> 
> do you want to add a field at the beginning of the message? (will the
> things you 
> are sending it to know what to do? or will it confuse their parsers?)
> 
> What I like to do is to send everything as a RFC3164 message, but with
> the body 
> of the message being a JSON structure. I set $!msg to the contents of
> $msg (if a 
> parser hasn't already done this for me), and I create a $!trusted branch
> that I 
> can add various metadata to (not just filename, but what input, what
> machine 
> sent it, what machine received it, what time each machine touched the
> message, 
> etc)
> 
> at the final destination, I have all that data available and can either
> use it, 
> or create a template that just writes out a RFC3164 style message with
> the 
> original message content.

Is there any reason why you prefer RFC3164 vs the later RFC5424
http://datatracker.ietf.org/doc/rfc5424 ?

Thanks
Dave
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Add the file name to syslog data

[David Lang]

any time you have a question like this, first log locally with the format 
RSYSLOG_DebugFormat so that you can see exactly what data you have where.


In this case, you will need to enable metadata in your imfile config, this will 
add the filename (and other information), but it doesn't do this by changing the 
message itself, it does this by creating file (under $!)


so you will need to look at the debug output and then decide how you are going 
to format the output message so that it contains the data.


do you want to add a field at the beginning of the message? (will the things you 
are sending it to know what to do? or will it confuse their parsers?)


What I like to do is to send everything as a RFC3164 message, but with the body 
of the message being a JSON structure. I set $!msg to the contents of $msg (if a 
parser hasn't already done this for me), and I create a $!trusted branch that I 
can add various metadata to (not just filename, but what input, what machine 
sent it, what machine received it, what time each machine touched the message, 
etc)


at the final destination, I have all that data available and can either use it, 
or create a template that just writes out a RFC3164 style message with the 
original message content.

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Add the file name to syslog data

[Joan via rsyslog]

Thanks for your reply deoren (due to some issue I had to read your mail on
the web archive
http://rsyslog-users.1305293.n2.nabble.com/Add-the-file-name-to-syslog-data-td7592722.html),
your approach is very neat unfortunately I need to send the data both to
graylog and syslog-ng. At this point we're not yet confident enough to rely
only on graylog (we'd need to do a proper multiserver install that's not
yet an option)
I will have to find a syslog compatible solution

2017-09-28 13:37 GMT+02:00 Joan :

> I am using rsylog to read (imfile) some logfiles and forward them to
> graylog in syslog format RFC5424.
> I would like to add an attribute with the filename, I explored doing it by
> modifiing the template (such as this
> ).
> But it will just add the information into the begin of the message,
> breaking some extractors I have in place.
> I though after reading about the RFC5424 that another option would be to
> add this data into the structured data fields (such as origin), but I
> couldn't find any example on how to add this information. Could you point
> me to some docs?
>
> Regards,
>
> Joan
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Add the file name to syslog data

[deoren]

On 9/28/2017 6:37 AM, Joan via rsyslog wrote:

I am using rsylog to read (imfile) some logfiles and forward them to
graylog in syslog format RFC5424.
I would like to add an attribute with the filename, I explored doing it by
modifiing the template (such as this
).
But it will just add the information into the begin of the message,
breaking some extractors I have in place.
I though after reading about the RFC5424 that another option would be to
add this data into the structured data fields (such as origin), but I
couldn't find any example on how to add this information. Could you point
me to some docs?

Regards,

Joan


Your post to the list reminded me of a task I've yet to start on, but 
hope to soon.


I too am currently forwarding from rsyslog to graylog using the 
RSYSLOG_SyslogProtocol23Format template and I also hope to extend the 
data I'm sending into Graylog to include metadata details like source 
filenames or inputs where a message is received.


Because I've recently been learning to use JSON payloads for shipping 
extra information (In a structured format), I looked up whether Graylog 
accepted log data in a JSON format. Turns out that not only does it 
support it, but it appears to accept it in a way that you can easily 
have Graylog create additional fields for you from supplied data.


I found a guide[1] for constructing an appropriate GELF template and 
plan to use that provided template as a starting point:


template(name="gelf" type="list") {
constant(value="{\"version\":\"1.1\",")
constant(value="\"host\":\"")
property(name="hostname")
constant(value="\",\"short_message\":\"")
property(name="msg" format="json")
constant(value="\",\"timestamp\":\"")
property(name="timegenerated" dateformat="unixtimestamp")
constant(value="\",\"level\":\"")
property(name="syslogseverity")
constant(value="\"}")
}

I say starting point because the Graylog documentation[2] covers 
extending the GELF payload to include additional data that Graylog will 
automatically treat as an additional field. That I believe refers to the 
metadata we wish to push into Graylog.


If this approach interests you, you may wish to read over the resources 
I've linked to below. Because I will be switching from the 
RSYSLOG_SyslogProtocol23Format template to a GELF TCP Graylog input, 
I've recorded in my TODO entry that I'll need to pay attention to these 
points:


* GELF TCP only supports uncompressed and non-chunked payloads. Each 
message needs to be delimited with a null byte (\0) when sent in the 
same TCP connection[2][3].


* Graylog is able to accept and parse RFC 5424 and RFC 3164 compliant 
syslog messages and supports TCP transport with both the octet counting 
or termination character methods[2][3].


In particular, because I've not changed the default value, I'm still 
using the "traditional" setting for the TCP_Framing parameter in my 
forwarding action from rsyslog to Graylog. According to what I've read 
from the sources below (this[4] one specifically mentions it), I should 
probably update that forwarding action to use the the octet-based TCP 
Framing option. This appears to be especially important with multi-line 
messages.


Not sure how that would apply if the message is entirely composed of a 
JSON structure vs a RFC 5424 or RFC 3164 compliant message.


[1] http://www.rsyslog.com/doc/v8-stable/tutorials/gelf_forwarding.html

[2] http://docs.graylog.org/en/2.3/pages/gelf.html#gelf-via-tcp

[3] http://www.rsyslog.com/doc/v8-stable/configuration/modules/omfwd.html

[4] 
https://selivan.github.io/2017/02/07/rsyslog-log-forward-save-filename-handle-multi-line-failover.html

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] Add the file name to syslog data

[Joan via rsyslog]

I am using rsylog to read (imfile) some logfiles and forward them to
graylog in syslog format RFC5424.
I would like to add an attribute with the filename, I explored doing it by
modifiing the template (such as this
).
But it will just add the information into the begin of the message,
breaking some extractors I have in place.
I though after reading about the RFC5424 that another option would be to
add this data into the structured data fields (such as origin), but I
couldn't find any example on how to add this information. Could you point
me to some docs?

Regards,

Joan
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.