Hrm. There's a *lot* of code and complexity in the LogstreamerInput. And not
because it's overengineered, IMHO, but because it's providing a lot of
functionality and solving a complicated set of problems. While I'd be thrilled
to have a SandboxInput replicating this functionality (mainly so it could be
used with Hindsight), I suspect that this is a much larger and more difficult
task than you might imagine. If you do end up going down that road, I suggest
you start with a tight, well-defined subset of LogstreamerInput's features, and
then work your way up from there.
I can't speak to the issues that you're having with trying to use `os.execute`
or `io.popen` to run an external sleep command. It's very possible that you're
bumping up against Heka bugs with the behaviour you're seeing. But that
approach smells a bit off to me, I'm not surprised you're having issues. You
might consider one of the other suggested mechanisms (see
http://lua-users.org/wiki/SleepFunction), especially using `socket.select`, or
writing a trivial C extension that exposes a sleep function. You also might try
using Hindsight instead of Heka to test your code, since that's a much lighter
code base and is likely to have fewer (or at least different) bugs.
You might already have this on your radar, but if I were working on this I'd
almost certainly be using coroutines
(http://lua-users.org/wiki/CoroutinesTutorial).
Finally, you say that the reason you're not using LogstreamerInput is because
you need to use Lua's `io` module in your decoding code. If you're okay with
building your own custom Heka binary, and you don't plan on using a
SandboxManagerFilter to support dynamic sandbox injection (i.e. you trust all
of the Lua code you'll be deploying) then you might consider rebuilding Heka
with support for the io module allowed in all sandboxes. The code that causes
the `io` module to not be available is here:
https://github.com/mozilla-services/heka/blob/v0.10.0b1/sandbox/lua/lua_sandbox.go.in#L63
Hope this helps,
-r
On 11/18/2015 05:40 AM, Kipras Mancevičius wrote:
Hey guys,
TL;DR any way to properly emulate LogStreamerInput behavior in Lua with
a SandboxInput ?
i'm having trouble trying to write a SandboxInput and since the docs
aren't the most helpful regarding that (no examples) - i thought i will
ask here.
Any directions/pointers for making a SandboxInput that would
continuously listen for new entries in a file? The reason why i'm not
using a LogStreamerInput input with a SandboxDecoder is that i need to
use the Lua 'io' module, and it's not available for SandboxDecoders only
for SandboxInputs.
The problems i'm having so far is:
* if the input does not have a "ticker_interval" set and
process_message() finishes - i get a "single run completed" entry in the
log and the input never restarts
* i tried using os.execute('sleep 1') / io.popen('sleep 1') after the
input is exhausted to wait for a bit and then recheck if there is any
new input, but:
os.execute('sleep 1') works for sleeping and listening for new input,
but stopping heka (Ctrl+C) does not work
io.popen('sleep 1') also works for sleeping and listening for new input,
but when trying to stop heka (Ctrl+C) on first Ctrl+C you get log
entries "Shutdown initiated.", "Stop message sent to input
'<your_input>'" but it does not shutdown (is there some way to handle
the "stop message" in the SandboxInput??) and on some subsequent Ctrl+C
it enters the infinte restart loop (read below)
* if the input has a "ticker_interval" - it either also never restarts
or restarts when you try to exit heka and enters a weird infinite loop
state, where it looks be infinitely restarting, heka is at 100% cpu and
you have to "kill -9" it to stop it. I think the infinite restart is
caused by the input crashing for some reason and then my global input
file position tracking is not updated and the input is restarted and so
it is processing the same input again and again. This behavior depends
on the set "ticker_interval", but i don't understand how those values
work, e.g.:
ticker_interval = 1000 : infinite restarting
ticker_interval = 1000000000 : infinite restarting
ticker_interval = 10000000000 : "single run completed" and never restarts
ticker_interval = 100000000000 : infinite restarting
This looks like some weird race conditions going on or smth..
I tried heka versions 0.9.2 and 0.10.0b1 and i'm getting the same
results on both.
Any way to properly emulate LogStreamerInput behavior in Lua with a
SandboxInput ?
_______________________________________________
Heka mailing list
[email protected]
https://mail.mozilla.org/listinfo/heka
_______________________________________________
Heka mailing list
[email protected]
https://mail.mozilla.org/listinfo/heka
