On 04/06/2015 11:49 AM, Tom Davis wrote:
Rob Miller writes:
>
> We've considered adding some sort of routing to the MultiDecoder, which would
allow you to look at input data and decide which decoder should receive it based
on arbitrary conditions, but that's not yet in place.
This would be really cool for my use case, too. Currently all of my decoders in
the MultiDecoder chain are implemented in Go so I'm not worried as much about
the performance implications (and load is quite low), but more advanced routing
would help as there are dependencies between the decoders. If the first fails,
trying any others is pointless because the message is already missing vital
data; it is up to the subsequent decoders to identify the failure, however,
since cascade_strategy=all has no short-circuit method.
What you're describing here is pretty much exactly why we recommend doing this
sort of thing entirely in Lua instead of in Go. Trying to express complex
conditional relationships using a declarative config format like TOML is never
going to be fun. A Turing complete language is a much better choice. If you
implemented your decoding logic in Lua instead of Go, it would be easy to write
a small amount of glue code to make sure that the decoding is handled
correctly, and that the correct actions are taken if a requisite step along the
way fails. Having us continue to add more and more complexity to the
MultiDecoder as more and more complicated use cases surface seems like a losing
battle.
Then again, I'm using Heka pretty outside its core use case of processing log
and analytics data.
I wouldn't say those are Heka's core use cases, they're just the paths that
have been most traveled, so far. Ideally Heka is useful in any situation where
you want to collect and ship data, possibly transforming it along the way.
I'm curious what your use cases are... mind sharing?
-r
_______________________________________________
Heka mailing list
[email protected]
https://mail.mozilla.org/listinfo/heka
