I'd be open to changing that to a configurable setting.
-r On 01/12/2016 01:11 AM, Eric LEMOINE wrote:
On Tue, Jan 12, 2016 at 9:30 AM, Eric LEMOINE <elemo...@mirantis.com> wrote: > Hi > > This is to report what may be an issue in Heka, where Heka consumes > CPU while there's nothing to do. > > My test-case is the following: LogStreamer with an rsyslog.lua decoder > reading logs from a number of static log files. After having processed > all the logs (from all the files) Heka still consumes CPU. > > On my 8-core laptop, with 500 static log file, Heka consumes 400% CPU > while processing the logs, which is expected given that I set max_proc > to 4 in heka.conf. BUT it still consumes 25% of CPU after having > processed all the logs, and this for ever. The more log files to be > processed the more consumed by Heka. So this is related to the number > of decoders Heka creates. > > Is this expected? Is this a bug in Heka. > > My test-case is here <https://github.com/elemoine/heka-cpu>, with > instructions on how to reproduce the problem. > > Thanks! > > PS: I'm happy to create a GitHub issue if you think it's appropriate. We figured the issue is related to the "check for more data interval" which is hard-coded to 250ms in the code [*]. So that value might not be appropriate when there are many log files to watch. 500 log files may not correspond to a realistic scenario. With 100 log files, which does sounds realistic to me, Heka still consumes about 8% of CPU on my 8-core laptop. Maybe 250ms is a bit too aggressive as a default value. Thoughts? [*] <https://github.com/mozilla-services/heka/blob/dev/plugins/logstreamer/logstreamer_input.go#L297> _______________________________________________ Heka mailing list Heka@mozilla.org https://mail.mozilla.org/listinfo/heka
_______________________________________________ Heka mailing list Heka@mozilla.org https://mail.mozilla.org/listinfo/heka
