Hi all I am in the process of re-designing our logging pipeline using centos/heka.
Up until now, servers would use Logstreamers with custom decoders to tail logs, assemble heka messages and AMQPOutput directly to two load-balanced RabbitMQ servers. Other heka instances would fetch the messages using AMQPInput and post them to ES using ESOutput. This has been running for a few months, processing millions of messages perfectly fine. Thanks, guys! :-) Now, I want to extend the pipeline for both internal and external systems. In order to achieve this, I've updated the pipeline so that: - "heka-producers" have many logstreamers and one single TCPOutput to stream *raw* messages to - "heka-aggregators" have one single TCPInput and many AMQPOutputs to queue *raw* messages - "heka-consumers" have many AMQPInputs with MultiDecoders to massage the data and one/many ESOutputs The idea is that all servers send raw logs in the Payload field of a heka message sent over TCP (leveraging buffering, output restart behaviour and TLS), heka-aggregators decouple producers and consumers by routing those logs to rabbitmq, and finally heka-consumers leverage MultiDecoders to apply arbitrary lua decoders to the deserialised heka messages (preserving the original Timestamp, Hostname etc set at the source). The following heka configs [1][2] used to work with 0.8.3. I have put together a small testing environment (same configs, mostly) and installed 0.10.0.b1. When the messages start flowing through the testing environment, I can confirm they are properly read and streamed to the heka-aggregators (as seen by LogOutput+RstEncoder), but in the heka-consumer side I get: 2015/08/31 14:33:40 Decoder 'system-rsyslog-mq1-input-system-rsyslog-multidecoder' error: Subdecoder 'system-rsyslog-decoder' decode error: Failed parsing: payload: 2015/08/31 14:33:40 :Timestamp: 1970-01-01 00:00:00 +0000 UTC :Type: :Hostname: :Pid: 0 :Uuid: :Logger: :Payload: :EnvVersion: :Severity: 7 The system-rsyslog-decoder fails at decoding the message as the payload that is presented to it is empty. The timestamps are also set to epoch. This suggests the problem lies in the deserialisation of the heka message posted to RabbitMQ. I suspect I have failed to align my configs with the splitter changes introduced around 0.9. I have experimented customising the splitter settings too - without any luck. The AMQPOutput and AMQPInput have also seen some changes, but I believe this is more of a PEBKAC than a bug. Any help would be greatly appreciated! [1] http://pastebin.com/LyDxrJtw [2] http://pastebin.com/UuDR8gVG -- "If you want to go fast, go alone. If you want to go far, go together." _______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
