I've put notes on what you've got here in-line below:
On 03/17/2015 10:44 AM, Hank Beatty wrote:
Hello,
Hi!
I'm trying to build some graphs from an ATS Server log file (custom
format). Here is a copy of the config that I came up with:
[ats_access_logs]
type="LogstreamerInput"
splitter = "TokenSplitter"
decoder = "ATS_transform_decoder"
log_directory = "/var/log/ats"
file_match = "custom_ats_psp6cdatsec04.log"
`file_match` is a regular expression value, so you'll save yourself some cycles by
escaping the period in your filename, either using
"custom_ats_psp6cdatsec04\\.log" or, even better, switching to single quotes
for TOML raw strings: 'custom_ats_psp6cdatsec04\.log'.
[ATS_transform_decoder]
type = "PayloadRegexDecoder"
match_regex = '(?P<UnixTimestamp>[\d]+\.[\d]+) chi=(?P<chi>\S+)
phn=(?P<phn>\S+) shn=(?P<shn>\S+) url=(?P<url>\S+) cqhm=(?P<cqhm>\w+)
cqhv=(?P<cqhv>\S+) pssc=(?P<pssc>\d+) ttms=(?P<ttms>\d+) b=(?P<b>\d+)
sssc=(?P<sssc>\d+) sscl=(?P<sscl>\d+) cfsc=(?P<cfsc>\S+)
pfsc=(?P<pfsc>\S+) crc=(?P<crc>\S+) phr=(?P<phr>\S+) uas=(?P<uas>\S+)'
#timestamp_layout= 'Dec 14 07:57:35'
[ATS_transform_decoder.message_fields]
Type = "ats_access"
host = "%phn%"
shn = "%shn%"
clientip = "%chi%"
Timestamp = "%UnixTimestamp%"
useragent = "%uas%"
uri = "%url%"
method = "%cqhm%"
status = "%pssc%"
crc = "%crc%"
phr = "%phr%"
version = "%cqhv%"
request_duration = "%ttms%"
We definitely recommend using Lua and LPEG for this type of parsing job. It's
much more composable, will perform a lot better, and provides a lot more
flexibility than the PayloadRegexDecoder. You can help test out an LPEG grammer
using our grammar tester (http://lpeg.trink.com/). There's a tutorial for
converting regex to LPEG in the Heka wiki
(https://github.com/mozilla-services/heka/wiki/How-to-convert-a-PayloadRegex-MultiDecoder-to-a-SandboxDecoder-using-an-LPeg-Grammar).
Also, if you find us in the #heka channel on irc.mozilla.org, we're often able
to help folks with debugging their grammars.
[ATSServer]
type = "SandboxFilter"
filename = "lua_filters/ats_graph.lua"
ticker_interval = 60
preserve_data = true
message_matcher = "Fields['Type'] == 'ats_access'"
[ATSServer.config]
sec_per_row = 60
rows = 1440
# anomaly_config = 'roc("HTTP Status", 2, 15, 0, 1.5, true, false)
roc("HTTP Status", 4, 15, 0, 1.5, true, false) mww_nonparametric("HTTP
Status", 5, 15, 10, 0.8)'
preservation_version = 1
[DashboardOutput]
ticker_interval = 60
I took the http_graph.lua and modified it slightly to fit the above. I
added some debug lines to the ats_graph.lua:
table.insert(dbg, "Exiting function process_message")
inject_payload ("txt", "debug", table.concat(dbg, "\n"))
I can't find where these are being written.
Any output from a SandboxFilter should show up in the DashboardOutput. Clicking on the
"Sandboxes" link should show you all of the sandbox filters and their outputs, clicking
on your filter's "debug" output should show you what's being generated. Is this not
happening?
From what I can tell the messages aren't making it to the
SandboxFilter. I'm thinking the message_matcher is wrong?
That may very well be. The best way to debug a message_matcher is to set up a
LogOutput (or, if the volume is too high, a FileOutput) using the same
message_matcher value to see what you're getting. The output should use an
RstEncoder, which will spit out a restructured text rendering of the entire
message for every message that the matcher catches.
If you do this and find that the message_matcher isn't catching the messages you want,
you can try a more general message_matcher, all the way up to `message_matcher =
"TRUE"`, which will catch everything. Once you're seeing the messages you want,
you can look at the ReST rendering of the message and compare that with your matcher,
it's usually pretty clear why things aren't working. Again, if you need help, during
California business hours we're usually able to help debug in the #heka channel. If the
mailing list works better for you and you want support debugging a matcher, it'd be
useful if you pasted your failing message_matcher and the full RstEncoder output for a
message that you think should be matching.
Any help would be greatly appreciated.
Hopefully this points you in the right direction.
If I can get this working I plan on trying to write a ATS log decoder
that I would be happy to share.
Ah, so you're just using the PayloadRegexDecoder to prototype, and plan on
developing an LPEG grammar when you've got data flowing? Great, sounds like
you're on the right track. :)
Thanks,
Hope it's useful,
-r
_______________________________________________
Heka mailing list
[email protected]
https://mail.mozilla.org/listinfo/heka
