I added a QueueMessage metric to the kafkaoutput plugin to debug idle message problems and found that QueueMessage hangs (notice the difference between Process and Queue counts): http://cl.ly/image/1i0I272K3B1R
...the metric increments one line after here: https://github.com/mozilla-services/heka/blob/kafka_output/plugins/kafka/kafka_output.go#L315 I recall QeueuMessage <https://github.com/Shopify/sarama/issues/70> being problematic back when I first started working w/ Sarama and ended up using SendMessage instead. My kafka plugin has been running for ~6 months now under SendMessage with much better stability (one of the heka binaries has been running since 6/17). Throughput is adequate, decode/encode are much more the bottleneck anyway. ...thought I'd share.
_______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
