I'm not sure exactly what is going on here, but I can provide you w/ some info that might help you debug further:
* Any packs that are in a decoder's input channel can only have come from the input pool, i.e. inputRecycleChan. I see ~40 packs tied up there. * Any packs that are in a match channel or input channel for a filter or output plugin could have come from either the input pool or the inject pool, although they can only have come from the inject pool if they were injected by some other filter plugin first. If a plugin's matcher only matches messages that you know came directly from an input plugin, then they have to have come from the input pool. Most of these are empty, but there are 60 packs sitting in the queues for http_metrics_filter. * Heka doesn't freeze the world while generating the report data, so it's possible that the data you're seeing doesn't represent a single point in time, which can cause the math to be weird. However, if Heka is truly wedged and no traffic is flowing, this is moot, since the state probably isn't changing at all during the entire reporting process. * It's possible for a filter or output to hold an arbitrary number of packs in its internal memory. These won't show up in the reporting at all, but neither will they be recycled. Without seeing the code that's running in your filters I can't say whether or not that's happening. I have to run into a meeting right now, hopefully this will help at least a bit, or at least help you come up with more specific questions. -r On 04/04/2016 12:09 PM, Eric LEMOINE wrote:
Hi We're running into a situation where Heka reports about "idle packs" and is "wedged". See the Heka diagnostics in <https://gist.github.com/elemoine/f533f674c514406fb6a320bd034d2568>. So the inject and input recycle channels are empty – no free packs. And the http_metrics_filter sandbox is blocked in an inject_message() call waiting for a free inject pack. We read other threads [*] discussing similar issues, but our case may be a bit different. The http_metrics_fliter gets messages from logstreamer inputs, and it injects messages that will be consumed by the influxdb_accumulator_filter. I think our case is different because the upstream filter (http_metrics_fliter), as opposed to the downstream filter (influxdb_accumulator_filter), is blocked. How can this be possible? And I really wonder where the inject packs are! The inject recycle channel is empty, so who holds the inject packs? There are 30 idle inject packs attributed to the influxdb_accumulator_filter, although that filter's match channel length is 0 (so is its input channel length). So I do not understand how this sandbox filter can have idle packs! And where are the remaining 70 inject packs? Any idea? In case this is relevant: we use buffering for the output plugins (with full_action drop), and no buffering for the filters. This is blocking us big time. Any insight is welcome. Thanks! [*] <https://mail.mozilla.org/pipermail/heka/2015-May/000586.html> _______________________________________________ Heka mailing list Heka@mozilla.org https://mail.mozilla.org/listinfo/heka
_______________________________________________ Heka mailing list Heka@mozilla.org https://mail.mozilla.org/listinfo/heka
