I actually have it passing byte slices regardless of buffering or not - it simplified the logic quite a bit to keep the same behavior between the modes, just SendRecord if not buffering, or QueueBytes if buffering. Retry isn't in there, and wasn't before. The previous behavior and new buffer disabled behavior just *discards *the buffer of records if the send fails. I'm hesitant to add retry since it means that the channel will block waiting for existing events in there to retry - I think the right answer should be "if you want retry, don't turn off buffering".
The other thing I noticed is that if the queue action is drop on full, it drops the newest events that don't fit, not the oldest - it might be worth it as part of the later plans for global buffering to add a "drop oldest" (or, as it's alternately known, a "circular") queue full action rather than just "drop new events on queue full". On Tue, Jan 13, 2015 at 11:29 AM, Rob Miller <[email protected]> wrote: > On 01/13/2015 07:27 AM, Tiru Srikantha wrote: > >> That's what the current behavior is actually - byte slices being passed >> back and forth in a channel. >> > > Right. Use of buffering avoids the need for this, but I'm thinking if > buffering isn't used then it still makes sense. > > I changed a bunch of stuff around so there >> is now a use_buffering option and it blindly sends if buffering is >> false, discarding failed batches. Buffering is on by default. >> > > Hopefully retry is still in effect even when buffering isn't. > > If you could give this ( >> https://github.com/highlyunavailable/heka/compare/feature/ >> bufferedelasticsearch >> ) another glance for code quality, that'd be great - also, let me know >> if there's some sort of other thing you'd like to see around this (unit >> tests, edits to changelog, etc.) before I submit a pull request. >> > > Will do. I'm in meetings most of today, so it'll probably be tomorrow > before I get to it. > > I did >> test this with an ES instance and it seems to perform well no matter how >> I kill the instance - it'll fail and then once the instance comes up >> just start sending again. The dashboard also reflects the correct number >> of processed messages now. >> > > Awesome, we're well on our way. Thanks for your efforts. :) > > -r >
_______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
