Timur - I hacked something similar together a while back to punt stats from both the heka.all-report and heka.memstats JSON outputs into OpenTSDB (though i guess it'd translate fairly trivially to things like InfluxDB): https://gist.github.com/hynd/98c4036cbc4a4f8a0f15
Each datapoint is a separate message, which might not be what you want, but it maybe of some use as a starting point? Cheers, -kieren On 16 November 2015 at 20:19, Timur Batyrshin <[email protected]> wrote: > Hi Rob, > > Parsing heka.all-report messages looks fine as well as having > DashboardOutput. > I was just hoping that there is already something ready for that :-) > > One more question related to my original issue but a bit away from topic > on performance: > As I’ve mentioned earlier I run heka agents on multiple boxes. > They send collect local data and send it via plain TCP to heka aggregator. > This central heka server then sends traffic further to storage via HTTP. > In my setup I see caches/logfiles on disk growing only on collectors but > not on aggregator. > > This means that if network connectivity issues appear on aggregator the > caches will fill up disks on end-boxes but not on aggregator, right? > Also looks like they'll even start dropping the messages. > So the question is: > Is there a way to make HttpOutput cache outgoing traffic in the same way > TcpOutput does? > Or is there any other way to add persistence to it (without running > external services)? > > Thanks, > Timur > > > On 16 Nov 2015 at 22:56:55, Rob Miller ([email protected]) wrote: > > Right now there are only two mechanisms for getting the information about > Heka's current operational state. One is that to which you're referring, > i.e. parsing the JSON from which the DashboardOutput generates its UI. You > can do this by fetching this through the dashboard's HTTP interface, > reading it from its location on the host's file system, or by setting up a > different filter and/or output that listens for the same messages that the > DashboardOutput is receiving, i.e. `Type == 'heka.all-report'`, where the > JSON will be the message payload. > > Unfortunately, these messages will only be generated if a DashboardOutput > is actually included as a part of your configuration. Ideally it'd be > possible to generate these messages without explicitly including a > DashboardOutput. It would also be nice to be able to generate the messages > with the relevant data encoded into message fields, rather than a big ol' > JSON blob in the payload. Neither of those are huge tasks, but they'd > require some amount of work to accomplish. > > The other mechanism for extracting this information is by sending a > SIGUSR1 signal to the running Heka process. This will cause the same data > that's included in the health report JSON to be spit out in a human > readable plain text format to Heka's console output. This was added so > admins would be able to see what's going on in cases where Heka's router is > wedged and messages aren't flowing, since in those cases clearly the report > messages aren't making it through. A DashboardOutput does not need to be > instantiated in this case, but of course monitoring and processing this > data is tricky. You'd have to tease it out of the other output that Heka is > emitting, and then parse the text format, which was designed for human > eyeballs and not really for machine parsing. > > Your best bet for this type of monitoring would probably be to write a > filter that listens for `heka.all-report` messages, parses the included > JSON, maintains a bit of state, and sends an alert if it notices that > certain tell-tale signs of slowness are persisting for more than a couple > of ticks. This requires having a DashboardOutput in your configuration, and > it's self-monitoring, so it will only work in cases where data is still > flowing in at least some capacity, but it should work for most of the cases > you're interested in. > > Hope this helps, > > -r > > > On 11/16/2015 09:00 AM, Timur Batyrshin wrote: > > Hi, > > > > Is there a way to measure Heka’s load except by parsing the output of > > DashboardPlugin in a special way? > > > > I’ve just hit a case when one of my router Heka’s had HttpOutput with > > non-existent host which severely decreased it’s performance (to > > ~10-20kb/sec which is no more than 1k messages/sec). > > I’d like to catch such cases in future as well as the cases when I need > > to think of some capacity increasing in other ways. > > > > I see only memory statistics in the standard Heka router messages while > > I think I’d wanted to watch for load in router and inputRecycleChan. > > Is there a standard way to do so? > > > > Thanks, > > Timur > > > > > > _______________________________________________ > > Heka mailing list > > [email protected] > > https://mail.mozilla.org/listinfo/heka > > > > > _______________________________________________ > Heka mailing list > [email protected] > https://mail.mozilla.org/listinfo/heka > >
_______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
