Getting caught up on backlog.... sorry for the delay.
On 10/18/2014 01:06 PM, Dieter Plaetinck wrote:
So I plan to do some experimenting with heka, but meanwhile I'ld like to hear from you. 1) does anyone have some empirical numbers on heka-as-statsd performance? in particular, when does it start dropping udp packets? what cpu % do you see (in particular can you spot spikes), and how many msg/s is it doing? 2) have you also noticed reading messages from the udp connection going slowly/maxing out cpu? i see heka calls read instead of readFromUDP like I do. I should look into that..
Unfortunately, I'm afraid we haven't done any of the detailed analysis you're describing of our statsd performance. I'd love to see it happen though, would you be interested in giving it a try?
3) does heka leverage multi-core properly for the statsd workload (receiving the packets, and doing the aggregations)?
Not sure what you mean by "properly", but it is true that the StatsdInput, which handles the UDP input and the parsing of the statsd string value, runs in a different goroutine than the StatAccumInput, which handles the aggregation, so those two tasks can run on separate cores. It would also be possible to use a regular UDP input, implement the statsd parsing in a decoder, and feed the stats into a StatAccumInput, which would mean the whole job would be spread across three different goroutines. But some testing would need to be done to see whether the benefits of spreading those jobs across multiple cores outweigh the drawbacks of the channel synchronization that is introduced with the use of more goroutines.
any other tips on replacing statsd with heka? is there a good tool to catch udp traffic and send it to both stats[daemon] and heka (so i can try the same production load on both)? there's xtables-addons kernel modules to do this, but i think our version of centos is too old. I quickly wrote up a go program to try to do this in userspace but i was hitting the same perf problem just calling net.UDPConn.readFromUDP in a loop.
No more specific tips, nope. As I said, though, I'd be happy to support an investigation into our performance as a statsd server, and to open bugs on any issues we find so we can improve. Sorry to not be much help, -r _______________________________________________ Heka mailing list Heka@mozilla.org https://mail.mozilla.org/listinfo/heka
