Hi Rob, Thanks! I was asking just about this. All is much clearer now. Now I need to try all this in code :-)
Thanks, Timur On Thu, Jul 9, 2015 at 9:25 PM, Rob Miller <[email protected]> wrote: > On 07/08/2015 11:26 PM, Timur Batyrshin wrote: > >> Hi all, >> >> I've just started to learn Heka and it looks really great. >> I have one question which doesn't look clear to me yet. >> >> How do we make dependencies/links between hosts? >> I need to do some calculations with different metrics (find delta, >> compare, etc) of both hostA and hostB at the same time. >> For example how do we compare number of messages passed through balancer >> to a sum of messages received by all backends (to route that to alert if >> delta is high)? >> > If you want to correlate data from two (or more) different sources, you'll > need to somehow route that data into the same Heka filter. > >> Looks like we need to do that using filters and looks like I need to >> write one for my needs but how should I deal with that? >> Can Lua sandboxes exchange data >> > Lua sandboxes can't directly access each other's memory spaces, but they > *can* exchange data. The mechanism for exchanging data is the Heka message. > > For instance, let's say there is a load balancer machine and three > separate back-end machines to which the load balancer is delegating. There > might be a Heka instance on each of those four machines, parsing log files > to gather information about the requests that each machine is handling. > > Each of these separate Heka instances can process the data it is > extracting from the local machine, possibly (although not necessarily) > aggregating it in some way. Then each of these Heka instances can forward > these messages (already parsed, in protobuf format) to another Heka > instance acting as an aggregator. The aggregator instance can have a filter > that matches on all of the messages from each of the separate hosts, so it > will receive all of them. This filter is then able to do any correlation or > other processing that is needed. > >> or do I have to keep several buffers >> inside my filter implementation and route different metrics to different >> buffers inside Lua or is there another good practice of doing that? >> > There isn't really any way to "route different metrics to different > buffers" inside of a filter. A filter can certainly manage multiple data > structures, so you could store data from one host in one data structure, > and store data from another host in a different one, and then correlate the > data between them. But it's up to you to do so in your code. > >> I'm just scratching surface now of how things work so general direction >> to which to look without going too much into details will be perfectly >> fine for me. >> > Hopefully this helps a bit? > > -r >
_______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
