The config that you cargo-culted from the docs is meant for an entirely different use case. That's meant to handle cases where you're receiving protocol buffer encoded Heka messages, each of which contains an Nginx access log line as the message payload. This would be useful in a case where one Heka is loading the log files but instead of parsing them it's sending them along in protobuf format to another Heka that's doing the parsing. The config below would be used on the listener.
If you want to see the decoding errors all you need to do is change your `log_sub_errors` setting from false to true. -r On 02/04/2015 04:19 AM, Victor Castell wrote:
I managed to get a working config but I want to understand what's going on: [syslog-decoder] type = "MultiDecoder" subs = ['nginx-access-decoder', 'rsyslog-decoder'] cascade_strategy = "first-wins" log_sub_errors = false In the nginx-access I'm deconding the corresponding access.log entries from my rsyslog and in the rsyslog decoder I'm capturing any other rsyslog entries and discarding them. That works well but in my first attempt I tried with the config extracted from the documentation: [shipped-nginx-decoder] type = "MultiDecoder" subs = ['ProtobufDecoder', 'nginx-access-decoder'] cascade_strategy = "all" log_sub_errors = true [ProtobufDecoder] I would rather this config than the previous one, because it can log me the errors of my nginx decoding. The problem is that when using the ProtobufDecoder the speed of decoding is really slow, and my nginx logs doesn't keep up with the current events in time, and it's always behind the current time. This doesn't happen with the rsyslog-decoder config, it parses the logs really fast. I thought it will be much faster using the internal ProtobufDecoder than a lua one but it's not the case. What's the reason for this? On Fri, Jan 30, 2015 at 11:31 AM, Victor Castell <[email protected] <mailto:[email protected]>> wrote: Didn't know of that! Life saver Thanks! El 30/1/2015 11:17, "Krzysztof Krzyżaniak" <[email protected] <mailto:[email protected]>> escribió: W dni pią 30 sty, 2015 o 10∶34 użytkownik Victor Castell <[email protected] <mailto:[email protected]>> napisał: > Hi! > > I have a centralized rsyslog formatted logfile and I'm > extracting nginx logs from there using heka and the nginx > access log decoder. > > The problem is that the parser also logs every other log > message out to heka.log. > > The volume of non nginx logs mixed in my rsyslog log is really > huge so heka.log file is growing like crazy (I have > logrotating before you ask) > > Is there a way to conditionally/intentionally suppress the > parsing errors of a given decoder? You probably want to use MultiDecoder which split nginx logs from the rest and use log_sub_errors = false in MultiDecoder section. eloy -- V _______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
_______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
