On 07/08/2015 11:26 PM, Timur Batyrshin wrote:
Hi all,
I've just started to learn Heka and it looks really great.
I have one question which doesn't look clear to me yet.
How do we make dependencies/links between hosts?
I need to do some calculations with different metrics (find delta,
compare, etc) of both hostA and hostB at the same time.
For example how do we compare number of messages passed through balancer
to a sum of messages received by all backends (to route that to alert if
delta is high)?
If you want to correlate data from two (or more) different sources, you'll need
to somehow route that data into the same Heka filter.
Looks like we need to do that using filters and looks like I need to
write one for my needs but how should I deal with that?
Can Lua sandboxes exchange data
Lua sandboxes can't directly access each other's memory spaces, but they *can*
exchange data. The mechanism for exchanging data is the Heka message.
For instance, let's say there is a load balancer machine and three separate
back-end machines to which the load balancer is delegating. There might be a
Heka instance on each of those four machines, parsing log files to gather
information about the requests that each machine is handling.
Each of these separate Heka instances can process the data it is extracting
from the local machine, possibly (although not necessarily) aggregating it in
some way. Then each of these Heka instances can forward these messages (already
parsed, in protobuf format) to another Heka instance acting as an aggregator.
The aggregator instance can have a filter that matches on all of the messages
from each of the separate hosts, so it will receive all of them. This filter is
then able to do any correlation or other processing that is needed.
or do I have to keep several buffers
inside my filter implementation and route different metrics to different
buffers inside Lua or is there another good practice of doing that?
There isn't really any way to "route different metrics to different buffers"
inside of a filter. A filter can certainly manage multiple data structures, so you could
store data from one host in one data structure, and store data from another host in a
different one, and then correlate the data between them. But it's up to you to do so in
your code.
I'm just scratching surface now of how things work so general direction
to which to look without going too much into details will be perfectly
fine for me.
Hopefully this helps a bit?
-r
_______________________________________________
Heka mailing list
[email protected]
https://mail.mozilla.org/listinfo/heka
