On 03/18/2015 04:44 AM, bruno binet wrote:
Thanks for your great support! You are right: the Severity variable was dropped by a previous sandbox filter which omitted to set Severity in the injected message:[...] inject_message({ Type = type_output, Timestamp = read_message('Timestamp'), Fields = data }) So to prevent this kind of neglecting in my lua sandboxes, I was wondering if I could only mutate some fields of an existing message instead of injecting a completely new message. Reading the sandbox documentation, it seems the "write_message" api function is the way to go: https://hekad.readthedocs.org/en/latest/sandbox/#write-message But when I try to use it instead of inject_message, I get the following error: "filters/encode_event.lua:31: attempt to call global 'write_message' (a nil value)"
At the end of the `write_message` section it's clearly documented that `write_message` is only available to decoders and encoders. It is deliberately not made available to filter plugins. Messages should never be mutated once they hit the router, for a couple of reasons, not least of which is that they might be concurrently processed by multiple output and/or filter plugins, and mutating the message would lead to race conditions. Because of this, SandboxFilters are able to create and inject new messages, but not mutate existing ones. Encoders support `write_message`, but it's a different implementation that transparently creates a new message under the hood (i.e. copy on write). This is to support certain specific use cases such as scrubbing personally identifiable information (PII) from messages on the way out.
Reading Heka source code, write_message seems to be used only in the tests, so it may not be available outside testing.
See above.
I think it would be useful to have this kind of api function available in the sandbox, or a function that would copy the current message to a lua table which can then be modified and reinjected later as a new message with the inject_message function.
This is a bit risky because if a filter tries to reinject a copy of a message that it just received, then that filter's message_matcher will match the copy. Heka doesn't allow such infinite routing loops and will shut down a sandbox filter that tries to inject to itself. To make this clear, we've erred on the side of not auto-copying data from the incoming message, since that forces you to think more carefully about the contents of the message you're creating. Generally, mutation is best done in the decoder step, which is explicitly for that purpose. Going forward, however, you will be able to approximate what you're asking for. I've recently landed a change on the dev branch that ensures that every message that hits the router will have a current, correct protobuf encoding of that message attached. In a sandbox filter, it'd then be possible to use `read_message(raw)` to get access to this binary encoding as a Lua string, followed by `decode_message(heka_protobuf_string)` to decode this into a Lua table. -r _______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
