On 11/16/2015 10:47 PM, Timur Batyrshin wrote:
On the other hand I was referring to 0.10.0b1 in which I still didn’t
see the buffering for HTTPOutput. Or do you mean it is still pending to
be merged?
Oops, that's just a documentation error. If you look at the "Common Filter Parameters" description
(http://is.gd/fFZJfL), you'll see some buffering related settings near the bottom, in the "New in
version 0.10" section. This links to the detailed buffering docs (http://is.gd/zvDwvQ). The "Common
Output Parameters" description for v0.10.0b1 (http://is.gd/wuRii2) doesn't mention these new buffering
settings, but it should. This has been resolved, as you can see in the dev version docs
(http://is.gd/r3xT0h), and will show up correctly in the v0.10.0b2 docs. Hoping to have that released later
this week.
> > Or is there any other way to add persistence to it (without running
> > external services)?
> The upcoming buffering is probably your best choice. There are
> hackish, fairly painful manual options, such as also writing your data
> out to a FileOutput, or maybe even using a SandboxFilter with
> `preserve_data = true` to hold on to a sliding window of the latest
> set of records, but there's no support for knowing what was missed and
> automatically retrying it, you'd have to basically cross-reference
> what arrived with what didn't and then manually extract the record
> from your backup storage and add it to the end data store by hand.
> Pretty painful. Sorry. :P
No problem, I’ll wait for the buffering to become available in mainline:
large lags in a single AWS region are not very likely so buffering on
collector hosts would be ok.
I was thinking of doing cross-datacenter data transfer which I’ll
probably hold for a while.
BTW, I’ve just though of one more use case for disk buffering: for
debugging purposes using heka-cat looks handy (vs LogOutput with
RstDecoder) and you’ll be able to use it when the buffers appear on disks.
Indeed, this is very useful. :)
-r
_______________________________________________
Heka mailing list
[email protected]
https://mail.mozilla.org/listinfo/heka
