Hi all,
Just wanted to send a note about a major breaking change that just
landed on the dev branch. We now have splitter plugins. They solve the
problem of deciding what the record boundaries are for a given input
stream, so Heka knows where to slice off a chunk of data that should be
decoded into a message.
Before the introduction of splitter plugins this was done by inputs on
an ad-hoc basis. A handful of our input plugins had
similar-but-slightly-different implementations, which was of course a
maintenance headache. The rest of them had no splitting support at all,
so any division of a single data payload had to happen downstream,
either in a decoder or a filter, neither of which were ideal for various
reasons.
The initial set of splitters we offer will be familiar to anyone who's
used Heka for a while:
* TokenSplitter (for single character delimited streams, replaces
`parser_type = "token"`)
* RegexSplitter (for when a single character isn't good enough, replaces
`parser_type = "regexp"`)
* NullSplitter (for when your incoming data is already split at natural
boundaries, e.g. one record per UDP packet, replaces the workaround of
`parser_type = "regexp"` and `delimiter = '$'`)
* HekaFramingSplitter (for Heka's protobuf streams, replaces
`parser_type = "message.proto"`)
Config documentation on splitters is available on ReadTheDocs:
http://hekad.readthedocs.org/en/dev/config/splitters/index.html
Most existing Heka configs will need to be edited, because any input
config section that specifies a "parser_type" value will no longer work.
Those settings should be replaced with the appropriate splitter setting
as described in the bulleted list above.
Additionally, you'll probably want to update any custom input plugins
you may have written to work with splitters. If you've depended on the
prior stream parsing code, you'll *have* to update your input because it
will no longer work. If not, your input might work, but it'd still
probably be a good idea to add splitter support so you will play nicely
with the rest of the infrastructure. I've detailed how the interaction
between inputs and splitters should work in the updated "Extending Heka"
documentation section:
hekad.readthedocs.org/en/dev/developing/plugin.html#inputs
I've also included what you need to know should you wish to write your
own splitters:
hekad.readthedocs.org/en/dev/developing/plugin.html#splitters
This is a pretty big change, so I'm hoping that plenty of folks will
test it out to help flush out the inevitable bugs. As always, please
send questions or comments to this mailing list, or come find us in the
#heka channel on irc.mozilla.org.
Thanks!
-r
_______________________________________________
Heka mailing list
Heka@mozilla.org
https://mail.mozilla.org/listinfo/heka
