Yeah, I'm looking at this as well. I don't like losing the bulk because that means it gets slow when your load gets high due to HTTP overhead. If I didn't care about bulk it'd just be a quick rewrite. The other problem I ran into around bulk operations on a buffered queue is that there are 4 points when you want to flush to the output:
1. Queued messages count equals count to send. 2. Queued message size with the new message added will exceed the max bulk message size, even if the count is less than max, due to large messages. 3. A timer expires to force a flush to the output target even if the count or max size hasn't been hit yet, to get the data into the output target in a timely manner. 4. Plugin is shutting down. I'm actually re-writing a lot of the plugins/buffered_output.go file in my local fork because of this and teasing out the "read the next record from the file pile" operation from the "send the next record" operation, so what I'm aiming for is something like a BulkBufferedOutput interface with a SendRecords(records [][]byte) method that must be implemented in lieu of BufferedOutput's SendRecord(record []byte) and some code that's shared between them to buffer new messages and read buffered messages. SendRecords would not advance the cursor until the bulk operation succeeded, as you might expect. I'll submit a PR once I finish. On Sat, Jan 10, 2015 at 6:57 PM, Denis Shashkov <[email protected]> wrote: > > >For this one case, though, an easier fix would be to have the ES output > retry, and/or if it fails to have the data that *would* have been sent to > ES to instead be written out to disk. > > I agree with Rob, that it's easier to patch ES output and add a retry. > > I've patched heka to add buffered output (like in TCPOutput plugin). It > perfectly works and we have simple monitoring of buffering and sending > performance. But... we completely lost ability to send bulk POSTs to ES > because BufferedOutput works only with single messages. Yes, we might use > flush_* parameters, but it's meaningless to keep messages in previous > buffer and to lost the whole bulk of them in ES output because of network > errors. > > BTW, Rob did you consider in your feature plans how to handle this > situation (single processing in BufferedOutput vs bulk sending outputs)? > > _______________________________________________ > Heka mailing list > [email protected] > https://mail.mozilla.org/listinfo/heka >
_______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
