Rob, Thanks for the quick reply.
Just tested with PayloadEncoder but got the similar results. But after reading your post I realised that: 1) My correct pipeline is: TcpInput -> TokenSplitter -> PayloadRegexDecoder -> ESJsonEncoder -> FileOutput 2) I should give it a go and remove TokenSplitter from the pipeline Indeed my EPS jumped to 21.6K (however I could no longer have the messages flowing through the pipeline as Heka no longer recognised them after their input. :D ) So on I guess the bottleneck is either in the Splitter or the Regex decoder (though the regex is a simple '^(?P<Payload>.*)' ) Cheers On Wed, Sep 23, 2015 at 2:42 PM, Rob Miller <[email protected]> wrote: > Almost certainly the slowest part of your pipeline there is the > ESJsonEncoder. What does the throughput look like if you replace that with > a PayloadEncoder? > > -r > > > On 09/22/2015 09:33 PM, Andre wrote: > >> Hi All, >> >> I was doing some performance tests around hekad with a simple test >> >> KVM VM running ubuntu, >> 2GB >> 4 core VM >> magnetic drives >> >> The test pipeline was the traditional vanilla pipeline similar to: >> >> TcpInput -> TokenSplitter -> ESJsonEncoder -> FileOutput >> >> The sample used was a stream of plain text TCP syslog generated by >> syslog-ng's loggen tool. >> >> Good news is that there's no message loss, bad news is that performance >> is somewhat lacklustre, while a normal rsyslogd doing similar work would >> do the same job at significantly higher rates, hekad was kind of stuck >> around 12K EPS. >> >> I started setting maxproc to 4, then tried playing with the poolsize >> (setting it to large value like 50000) but the only consequence of that >> was heka consuming more memory, EPS continue more or less the same. >> >> Has anyone exceeded this performance with hekad under similar pipelines >> (i.e. TcpInput) and HW conditions (small VMs)? >> >> If yes, mind sharing a bit on how the hekad was configured? >> >> Kind regards >> >> >> _______________________________________________ >> Heka mailing list >> [email protected] >> https://mail.mozilla.org/listinfo/heka >> >> >
_______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
