That seems to be an awful lot of work for something that you could probably do fairly easily with Lua in the Sandbox. Check out the InfluxDB Batch sandbox filter.
https://github.com/mozilla-services/heka/blob/dev/sandbox/lua/filters/influx_batch.lua -- Robison Jacka / OpsEng / Weebly On Tue, Feb 16, 2016 at 4:42 PM, Eli Flesher <efles...@lyft.com> wrote: > Hi Everyone, > > I’m using Heka to ingest logs from a large cluster of boxes and then move > them off to an API. I have a working prototype, but right now I’m using the > LogStreamer and default TokenSplitter to create a message per log line. > > This is not optimal, given that it creates a large number of messages and > thus API calls on the support system. > > I’d like to buffer up to 1MB of log data and create messages in 1MB > chunks. Reading over the Extending heka documentation, it seems that this > could be done with either a Splitter plugin or a Filter plugin. > > The ides of the splitter would be similar to the TokenSplitter. It would > check to see if the byte slice of data passed in is over the buffer size > and select a buffer’s worth for a message from the slice and ‘read’ that > much. It would otherwise indicated 0 bytes read as the TokenSplitter does. > I’m wondering if I would need to tweak any of the global configuration > options to make this work (e.g. max_message_loop, plugin_chansize or > max_message_size). > > Alternatively, I’m thinking of implementing a filter that collects these > messages in a buffer and flushes the buffer when the desired size is > reached. The problem with this is I’ll have multiple log streams and I > wouldn’t want to cross the streams. Also, as much as possible, I’d like to > preserve the order of lines within a single chunk (messages themselves are > encoded with a timestamp for later reassembly). > > Thoughts on either approach? > > > Thanks, > > > Eli > -- > — > *Elijah Flesher* | *Lyft* <http://lyft.me/> | *Software Engineer* > 206.661.4697 | @eliflesher > > _______________________________________________ > Heka mailing list > Heka@mozilla.org > https://mail.mozilla.org/listinfo/heka > >
_______________________________________________ Heka mailing list Heka@mozilla.org https://mail.mozilla.org/listinfo/heka
