On Tue, Jun 30, 2015 at 11:56 AM Rob Miller <[email protected]> wrote:
> Encoders don't emit message structs, they emit byte streams. If a Lua > encoder uses the `inject_message` API call then it will emit the protobuf > encoding for the full message, but if the `inject_payload` API is used then > it will emit the string contents of the provided payload. > > oh, this is interesting. so `inject_message` in an encoder can serve to scribble on the message just before it hits the wire, but as a protobuf? > In other words, you're free to generate and emit any arbitrary JSON you > want. Your code should do the following: > > - Call read_message to extract any needed data from the incoming message > - Create a Lua table matching the JSON structure you want to generate > - Use the `cjson` module to encode the table to a JSON string > - Call zero or more `add_to_payload` calls, followed by an > `inject_payload` call to emit whatever strings you want to emit > > Here's the source code for the ESPayloadEncoder: > > > https://github.com/mozilla-services/heka/blob/versions/0.9/sandbox/lua/encoders/es_payload.lua > > That doesn't do any JSON encoding, b/c it assumes that the payload coming > in will already contain the raw JSON that ElasticSearch expects. It does, > however, show the use of read_message, add_to_payload, and inject_payload. > As an added bonus it shows you how to read data from the config file, and > how to use the `elasticsearch` Lua module we provide to generate the ES > Bulk API index headers. > > -r > > > On 06/30/2015 11:10 AM, Lewis, Steven wrote: > > Hello heka people! > > We have a kibana/elasticsearch stack already running (with custom json > > formats), and I’d like to massage the message coming out of some JSON > > inputs we have to not exactly follow the Heka message format. > > Specifically, I’d like the JSON contained in the “@message” value to > > move up a level, so that _source directly contains my relevant keys, > > like so: > > {“_source”: { > > “@uuid”: <guid here>, > > “@timestamp”: <timestr here>, > > “@ID”: <my custom guid>, > > “@status”: <custom status> > > } > > I was thinking at first that I could create a lua encoder, but looking > > at the API I don’t see anything where I can modify the output directly. > > I can modify the payload, or I can modify the message, but I can’t go > > outside the schema of the Heka message (at least that’s my impression). > > Do I need to create a Sandbox Output to do this? Or copy the > > ElasticSearch go output and tweak it? Hopefully my question makes sense. > > We can get what we want if we use logstash, but we love go and heka > > seems awesome. > > Thanks for any guidance! > > Steve Lewis > > Sr. Software Engineer > > PETNET Solutions, Inc. > > This message and any attachments are solely for the use of intended > > recipients. The information contained herein may include trade secrets, > > protected health or personal information, privileged or otherwise > > confidential information. Unauthorized review, forwarding, printing, > > copying, distributing, or using such information is strictly prohibited > > and may be unlawful. If you are not an intended recipient, you are > > hereby notified that you received this email in error, and that any > > review, dissemination, distribution or copying of this email and any > > attachment is strictly prohibited. If you have received this email in > > error, please contact the sender and delete the message and any > > attachment from your system. Thank you for your cooperation > > > > > > _______________________________________________ > > Heka mailing list > > [email protected] > > https://mail.mozilla.org/listinfo/heka > > > > _______________________________________________ > Heka mailing list > [email protected] > https://mail.mozilla.org/listinfo/heka >
_______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
