On Wed 05 Feb 2014 10:53:17 AM PST, Lev Brouk wrote:
Hi
I have a rotating log file that I am trying to get into heka, using
Logstreamer and PayloadJsonDecoder and have a few questions about it.
1. The rotation naming scheme:
/somelogdir/
(oldest)
mylogfile.log_19700101..something.old
mylogfile.log_2040201..something.old
mylogfile.log_2040202..something.old
....
mylogfile.log
(newest)
I set this up as
[mylog_Streamer]
Type = "LogstreamerInput"
log_directory = "/somelogdir"
file_match =
'mylogfile\.log(_(?P<Year>\d{4})(?P<Month>\d\d)(?P<Day>\d\d)\.(?P<stuff>.*)\.(?P<Ftype>old))?'
priority = ["^Ftype","Year","Month","Day"]
[mylog_Streamer.translation.Ftype]
old = 1
Is there a better way to make the files sort the "right" way?
(Ideally, I'd provide the default values for the missing captures, or
at least specify +- infinity for the sort order)
Nope, you've hit it on the head here. We've already realized that we
need to support default values for the missing captures, see
http://is.gd/LiROzx. That will probably land next week.
2. Decoding.
Each line in the file is formatted as:
<version-stamp> <json_data>
e.g.
2 { ...some-json...}
Short of adding a regex extract capability to the PayloadJsonDecoder,
is there an existing way of parsing the JSON into the message fields?
3. Timestamp.
In the above JSON the timestamp is an integer, which represents the
number of milliseconds since the epoch, UTC. I do not believe that it
can be handled by a go time format. Any suggestions on how to best get
it converted in?
To resolve both of these issues, you'll want to stop using the
PayloadJsonDecoder and instead use a SandboxDecoder w/ some Lua code to
do the transformation that you want. We expose a JSON parsing library
for you to use, here are some examples of how it might be used:
http://hekad.readthedocs.org/en/latest/sandbox/json_payload_transform.html.
This is not only infinitely more flexible than using the Go-based JSON
parsing in the PayloadJsonDecoder, it's also considerably faster,
believe it or not.
Note that those example decoder snippets linked above use the
`inject_message` API call, which scraps the message that was triggering
the decode and replaces it with a new message populated by (in this
case) the parsed JSON data. If you want to mutate the message that is
being processed instead of replacing it altogether you can use
`write_message` to modify specific message fields. More info in the
docs: http://is.gd/S56u1b.
Hope this helps!
-r
_______________________________________________
Heka mailing list
[email protected]
https://mail.mozilla.org/listinfo/heka
