Re: [heka] KafkaOutput: Expected behavior during a kafka broker outage

2015-10-22 Thread Andre
All, I've put together the code to address this issue. I basically followed the strategy used on TCP/HTTP/IRC output plugins together with Sarama's SyncProducer. https://github.com/trixpan/heka/commit/c5b455f80dd89216b3ff5d6da7880bb6522b6ee7 It works greatly and Heka is capable of surviving a f

[heka] Lua Decoder Parse Error

2015-10-22 Thread Justin Vogt
Hello Heka Community, I’m getting a weird error… I’ve written a custom decoder in Lua, and it works find when I test it on the LPEG grammar tester, but when I try to run it in Heka, I continually get a “Decoder error, failed parsing” message. I’ve tried just about everything I can think of and hav

Re: [heka] Lua Decoder Parse Error

2015-10-22 Thread Nathan Williams
This looks like a pretty straight-forward syslog message, have you tried the rsyslog decoder[1]? If you need to do further processing past the initial rsyslog decoding, a custom decoder that still uses the syslog module to do the initial grammar parsing before further processing works pretty well,

Re: [heka] Lua Decoder Parse Error

2015-10-22 Thread Michael Trinkala
It is probably not the grammar failing, I am betting you return -1 somewhere else in the decoder. Can you share the code? Trink On Wed, Oct 21, 2015 at 11:23 AM, Justin Vogt wrote: > Hello Heka Community, > > I’m getting a weird error… I’ve written a custom decoder in Lua, and it > works find

Re: [heka] Lua Decoder Parse Error

2015-10-22 Thread Vogt, Justin
Here’s the whole lua file: local l = require "lpeg" local dt = require "date_time" local sp = l.space l.locale(l) local pri = l.P"<" * l.Cg(l.R"09"^0, "pri") * l.P">" local logtime = l.Cg(dt.build_strftime_grammar("%b %d %X"), "logtime") * sp local hostname = l.Cg((1 - sp)^1, "hostname") * sp lo

Re: [heka] Lua Decoder Parse Error

2015-10-22 Thread Michael Trinkala
The code is failing on the inject_message line. The message Fields table must follow this: https://hekad.readthedocs.org/en/latest/sandbox/index.html#lua-message-hash-based-field-structure schema. Also, you could trap the error from the pcall and return it with the status code to avoid some of th

Re: [heka] Lua Decoder Parse Error

2015-10-22 Thread Vogt, Justin
Now I’m not quite sure what I’m missing… I keep getting this error: inject_message() could not encode protobuf - unsupported type: nil I’m trying the bare minimum of msg here: local l = require "lpeg" local dt = require "date_time" local sp = l.space l.locale(l) local pri = l.P"<" * l.Cg(l.R"0

Re: [heka] Lua Decoder Parse Error

2015-10-22 Thread Vogt, Justin
Alright I got it… my “logtime” table below was causing formatting issues. Once I removed that from the capture then it started working. Thanks for looking into this! A great learning experience for me. - Justin On 10/22/15, 3:11 PM, "Heka on behalf of Vogt, Justin" wrote: >Now I’m not