On Mon, May 12, 2014 at 2:50 PM, Rob Miller <[email protected]> wrote:
> Hi! > > Just wanted to give everyone a heads up that a significant change will be > landing on the dev branch of Heka soon, to be included in the eventual 0.6 > release. > > This change will introduce a new "Encoder" plugin type. Encoders are the > inverse of the already existing Decoder plugin type. Decoders are used by > Input plugins to convert arbitrary input data into a Heka message struct, > Encoders are used by Output plugins to convert Heka message structs into > arbitrary output data. > > We realized this was a necessary abstraction when we saw that various > different Output plugins were implementing their own ways to manage the > serialization. For instance, currently: > > - FileOutput supports a 'format' config option which can be one of 'text', > 'json', or 'protobufstream'. The 'text' option includes only the message > payload. The 'json' format contains all of the message fields, but is an > inflexible format. > > - LogOutput supports a 'payload_only' boolean config option. If true, then > the message payload will be written to stdout. If false, then a custom, > inflexible text rendering of the message data will be generated. > > - ElasticSearchOutput supports a 'format' config option which can be one > of 'clean', 'logstash_v0', 'payload', or 'raw'. All of them generate JSON > in a specific format, except 'payload', which presumes that the message > payload already contains the JSON that you want to send to ElasticSearch. > > - TcpOutput has no flexibility, it only generates Protocol Buffer encoded > message streams. > > The introduction of Encoder plugins means all of these one-off > serialization strategies can go away. Instead, you'll add an Encoder config > section to your TOML config, and then you'll refer to configured Encoder > sections from your Output config sections. So what would have been this: > > [LogOutput] > payload_only = true > > Instead will be this: > > [PayloadEncoder] > > [LogOutput] > encoder = "PayloadEncoder" > > The initial code will include three encoders: ProtobufEncoder (generates > Heka's native protocol buffer streams), PayloadEncoder (extracts message > payload), and SandboxEncoder (lets you use Lua code to extract data from a > message and generate whatever output you want). There may be more coming in > the future, but really our hope is that the SandboxEncoder will meet most > of your needs. > > Also, initially the TcpOutput, LogOutput, and FileOutput have been > modified to use Encoders instead of their previous mechanisms. TcpOutput > defaults to use of ProtobufEncoder, which exactly matches the previous > behavior, so no config changes should be necessary. If you're using > LogOutput or FileOutput, however, when you upgrade you'll need to modify > your config to include appropriate Encoder plugins and to make sure they're > being used by your outputs. > > Anyone interested in digging in to the code can take a look at the open > pull request at https://github.com/mozilla-services/heka/pull/838. It's > currently awaiting code review, which might result in further small > revisions, but we definitely expect to land this on dev over the next few > days. I'll send another note out when it lands. > > If you made it this far, wow, I'm impressed! Thanks for your attention, > hope the Encoder plugins work for you, and please let us know if you have > any questions or issues. > > Thanks! > > -r > I've got a bunch of code that I'll be able to dump greatly simplifying both the build process and the configuration needed to manage all that extra code. Looking forward to 0.6! _______________________________________________ > Heka mailing list > [email protected] > https://mail.mozilla.org/listinfo/heka >
_______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
