On 07/09/2015 01:35 PM, Ali wrote:
Hi, all.
So, I finally got a custom Lua decoder to parse events coming in via
TcpInput. On the other end, it's just nxlog watching a bunch of log
files and then sending the new events to Heka via TCP. The log files
are all named SystemOut.log and are located throughout a directory tree,
one file per server.
Once the data gets to Heka, Heka sends it to Elasticsearch and I am
viewing the results in Kibana.
I started examining the data and found that I had no way of determining
which SystemOut.log file a given event came from. :-) I have been
playing with nxlog, rewriting the message on that end and then letting
Heka parse out the SystemOut.log path that I'm tacking onto the end of
the message. That seemed really wrong, though. I thought, if I'm doing
this -- modifying the data at all -- I should really be doing it in a
structured way, like using nxlog's JSON output and then adding an
attribute for "filepath" or something.
I'm all for structured logging, but I wouldn't be too upset about
tacking the info you need on to the end of a log line. If I were
evaluating these options, I'd be looking at performance and my scaling
needs. I'd try both the plain text version parsed with LPEG *and* the
JSON version parsed with cjson and see which gives better performance.
Which begs the question, what is the right way to handle this on the
Heka side?? As I said, I already wrote the Lua decoder that parses out
the payload and the funny timestamp so I'm thinking I'll still have to
use it. But should I use that to parse out the JSON? It doesn't look
like the Lua standard library has JSON parsing. Should I use one
decoder to parse the JSON and the custom Lua decoder to parse out the
timestamp? Seems like this would be simple to do in a Go plug-in.
As David said, the sandbox exposes the cjson module, which makes parsing
arbitrary JSON trivial. But LPeg is fast, often faster than JSON
parsing, so it might not be worth it.
-r
_______________________________________________
Heka mailing list
[email protected]
https://mail.mozilla.org/listinfo/heka
