Re: [heka] heka wedged

2016-03-30 Thread Timur Batyrshin
Hi Ramin, When heka is running it emits HeapAlloc, HeapSys and few other metrics which you can graph or alert based on them. I've seen heka freezing only when I send TERM signal to it and some Lua plugin fails to stop correctly or something like that. May be someone else will give you insights on

Re: [heka] heka wedged

2016-03-30 Thread Ramin Ali Dousti
Hi Timur, 1) wedged: In my case, my hekad gets wedged for no reason (or I think i cannot pinpoint as to why) 2) logging Yes, I could do a LogOutput (which I'm doing when I'm troubleshooting) but I was looking for some kind of application logging with different severity so that i could up or down

Re: [heka] heka wedged

2016-03-30 Thread Rob Miller
Heka can get wedged when there's a deadlock in the pipeline, usually related to pack exhaustion. For example, a filter might block because it's waiting for an empty pack with which to inject a new message, but there are no packs available because they're all tied up in the input channels for th

Re: [heka] heka wedged

2016-03-30 Thread Ramin Ali Dousti
Hi Rob, thanks for the response/info. Yes, as you might remember I implemented your heartbeat suggestion months ago and am sending heartbeats of a chain of hekad's to influx and am using kapacitor's deadman watch to catch the lack of heartbeat and it's working perfectly fine. The problem I'm tryi