Thanks for the explanations and the work around.
On 18 March 2015 at 17:50, Rob Miller <[email protected]> wrote:
> On 03/18/2015 04:44 AM, bruno binet wrote:
>
>> Thanks for your great support!
>> You are right: the Severity variable was dropped by a previous sandbox
>> filter which omitted to set Severity in the injected message:
>>
>> [...]
>> inject_message({
>> Type = type_output,
>> Timestamp = read_message('Timestamp'),
>> Fields = data
>> })
>>
>> So to prevent this kind of neglecting in my lua sandboxes, I was
>> wondering if I could only mutate some fields of an existing message
>> instead of injecting a completely new message.
>> Reading the sandbox documentation, it seems the "write_message" api
>> function is the way to go:
>> https://hekad.readthedocs.org/en/latest/sandbox/#write-message
>>
>> But when I try to use it instead of inject_message, I get the following
>> error:
>> "filters/encode_event.lua:31: attempt to call global 'write_message' (a
>> nil value)"
>>
> At the end of the `write_message` section it's clearly documented that
> `write_message` is only available to decoders and encoders. It is
> deliberately not made available to filter plugins. Messages should never be
> mutated once they hit the router, for a couple of reasons, not least of
> which is that they might be concurrently processed by multiple output
> and/or filter plugins, and mutating the message would lead to race
> conditions.
>
> Because of this, SandboxFilters are able to create and inject new
> messages, but not mutate existing ones. Encoders support `write_message`,
> but it's a different implementation that transparently creates a new
> message under the hood (i.e. copy on write). This is to support certain
> specific use cases such as scrubbing personally identifiable information
> (PII) from messages on the way out.
>
> Reading Heka source code, write_message seems to be used only in the
>> tests, so it may not be available outside testing.
>>
> See above.
>
>> I think it would be useful to have this kind of api function available
>> in the sandbox, or a function that would copy the current message to a
>> lua table which can then be modified and reinjected later as a new
>> message with the inject_message function.
>>
> This is a bit risky because if a filter tries to reinject a copy of a
> message that it just received, then that filter's message_matcher will
> match the copy. Heka doesn't allow such infinite routing loops and will
> shut down a sandbox filter that tries to inject to itself. To make this
> clear, we've erred on the side of not auto-copying data from the incoming
> message, since that forces you to think more carefully about the contents
> of the message you're creating. Generally, mutation is best done in the
> decoder step, which is explicitly for that purpose.
>
> Going forward, however, you will be able to approximate what you're asking
> for. I've recently landed a change on the dev branch that ensures that
> every message that hits the router will have a current, correct protobuf
> encoding of that message attached. In a sandbox filter, it'd then be
> possible to use `read_message(raw)` to get access to this binary encoding
> as a Lua string, followed by `decode_message(heka_protobuf_string)` to
> decode this into a Lua table.
>
> -r
>
_______________________________________________
Heka mailing list
[email protected]
https://mail.mozilla.org/listinfo/heka
