2015-04-16 19:34 GMT+02:00 Rob Miller <[email protected]>: > On 04/16/2015 04:48 AM, Etienne Champetier wrote: > >> Hi, >> >> I'm using heka to read logs with logstreamer (nginx access log) and ship >> them to another heka via TcpOutput. >> I'm new to heka (and this ML) so here are my questions: >> >> Is bufferedoutput (to disk) mandatory? (maybe i'm not understanding the >> code correctly/missed a config) >> > TcpOutput always runs the messages through a disk buffer, yes. This might > be optional in the future, but it's true for now. > >> In my case it produces useless IO, ie read logs from disk, split them, >> parse them, write them to disk, read them again and send them via tcp. >> > In our experience the performance impact from running the messages through > a disk buffer is small enough, and TCP connections are brittle enough, that > it wasn't worth introducing more complexity into the code to support both > options. We have plans to change the way buffering works entirely, however, > so it's optionally available to all plugins, this will change your options > with regard to the TcpOutput. >
ok cool (issue https://github.com/mozilla-services/heka/issues/1378 i guess) Maybe you should add a note in the documentation Is there plan for memory buffer output (buffered chan, ...). >> > There haven't been plans thus far, no. Buffered channels are always in > play, but there's no retry, messages that don't get through will be > dropped. I'm definitely open to considering in memory buffering options, > but not until after we get all of the details of the disk buffering > situation worked out. > >> Output buffer shrink only every ticker_interval, which is 300sec, isn't >> it too much by default? (on first run of heka with big input logs buffer >> can grow a lot) >> > Whether or not this is a reasonable default depends on how much data is > flowing through, and how precious a commodity disk space is. I'm open to > lowering it, however, and explaining the implications a bit better in the > docs. > now that i think a bit more about it, instead of time based rotation maybe this should be size based rotation (i've seen a buffer of ~500Mo) Also what are the drawbacks of rotating too often? queue_full_action = block doesn't recover (you have to restart heka) or >> i'm not patient enough? >> > No, it's not you... I've done some testing and have been able to > reproduce. I'm currently working on a fix, more soon. > Thanks a lot (i've opened an issue https://github.com/mozilla-services/heka/issues/1484) > -r > >
_______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
