Encoders don't emit message structs, they emit byte streams. If a Lua encoder
uses the `inject_message` API call then it will emit the protobuf encoding for
the full message, but if the `inject_payload` API is used then it will emit the
string contents of the provided payload.
In other words, you're free to generate and emit any arbitrary JSON you want.
Your code should do the following:
- Call read_message to extract any needed data from the incoming message
- Create a Lua table matching the JSON structure you want to generate
- Use the `cjson` module to encode the table to a JSON string
- Call zero or more `add_to_payload` calls, followed by an `inject_payload`
call to emit whatever strings you want to emit
Here's the source code for the ESPayloadEncoder:
https://github.com/mozilla-services/heka/blob/versions/0.9/sandbox/lua/encoders/es_payload.lua
That doesn't do any JSON encoding, b/c it assumes that the payload coming in
will already contain the raw JSON that ElasticSearch expects. It does, however,
show the use of read_message, add_to_payload, and inject_payload. As an added
bonus it shows you how to read data from the config file, and how to use the
`elasticsearch` Lua module we provide to generate the ES Bulk API index headers.
-r
On 06/30/2015 11:10 AM, Lewis, Steven wrote:
Hello heka people!
We have a kibana/elasticsearch stack already running (with custom json
formats), and I’d like to massage the message coming out of some JSON
inputs we have to not exactly follow the Heka message format.
Specifically, I’d like the JSON contained in the “@message” value to
move up a level, so that _source directly contains my relevant keys,
like so:
{“_source”: {
“@uuid”: <guid here>,
“@timestamp”: <timestr here>,
“@ID”: <my custom guid>,
“@status”: <custom status>
}
I was thinking at first that I could create a lua encoder, but looking
at the API I don’t see anything where I can modify the output directly.
I can modify the payload, or I can modify the message, but I can’t go
outside the schema of the Heka message (at least that’s my impression).
Do I need to create a Sandbox Output to do this? Or copy the
ElasticSearch go output and tweak it? Hopefully my question makes sense.
We can get what we want if we use logstash, but we love go and heka
seems awesome.
Thanks for any guidance!
Steve Lewis
Sr. Software Engineer
PETNET Solutions, Inc.
This message and any attachments are solely for the use of intended
recipients. The information contained herein may include trade secrets,
protected health or personal information, privileged or otherwise
confidential information. Unauthorized review, forwarding, printing,
copying, distributing, or using such information is strictly prohibited
and may be unlawful. If you are not an intended recipient, you are
hereby notified that you received this email in error, and that any
review, dissemination, distribution or copying of this email and any
attachment is strictly prohibited. If you have received this email in
error, please contact the sender and delete the message and any
attachment from your system. Thank you for your cooperation
_______________________________________________
Heka mailing list
[email protected]
https://mail.mozilla.org/listinfo/heka
_______________________________________________
Heka mailing list
[email protected]
https://mail.mozilla.org/listinfo/heka
