Hi Eli, A bit offtopic: when you need to offload messages to API make sure you can dump the whole message in a single run: output encoders don't support sending out several messages from a single heka message (only filters and decoders can do that).
Regards, Timur On Wed, Feb 17, 2016 at 3:42 AM, Eli Flesher <efles...@lyft.com> wrote: > Hi Everyone, > > I’m using Heka to ingest logs from a large cluster of boxes and then move > them off to an API. I have a working prototype, but right now I’m using the > LogStreamer and default TokenSplitter to create a message per log line. > > This is not optimal, given that it creates a large number of messages and > thus API calls on the support system. > > I’d like to buffer up to 1MB of log data and create messages in 1MB > chunks. Reading over the Extending heka documentation, it seems that this > could be done with either a Splitter plugin or a Filter plugin. > > The ides of the splitter would be similar to the TokenSplitter. It would > check to see if the byte slice of data passed in is over the buffer size > and select a buffer’s worth for a message from the slice and ‘read’ that > much. It would otherwise indicated 0 bytes read as the TokenSplitter does. > I’m wondering if I would need to tweak any of the global configuration > options to make this work (e.g. max_message_loop, plugin_chansize or > max_message_size). > > Alternatively, I’m thinking of implementing a filter that collects these > messages in a buffer and flushes the buffer when the desired size is > reached. The problem with this is I’ll have multiple log streams and I > wouldn’t want to cross the streams. Also, as much as possible, I’d like to > preserve the order of lines within a single chunk (messages themselves are > encoded with a timestamp for later reassembly). > > Thoughts on either approach? > > > Thanks, > > > Eli > -- > — > *Elijah Flesher* | *Lyft* <http://lyft.me/> | *Software Engineer* > 206.661.4697 | @eliflesher > > _______________________________________________ > Heka mailing list > Heka@mozilla.org > https://mail.mozilla.org/listinfo/heka > >
_______________________________________________ Heka mailing list Heka@mozilla.org https://mail.mozilla.org/listinfo/heka
