Hi, thanks for all the info. I'll look into the options again. Most probably going to recompile Heka and enable the "io" module.
As for coroutines - i did try to use them as they seemed to be the way to go for implementing this, however they didn't work - it seems that "yield()' is not supported in the Heka Lua sandbox (it complains about an undefined global). On Thu, Nov 19, 2015 at 12:16 AM, Rob Miller <[email protected]> wrote: > Hrm. There's a *lot* of code and complexity in the LogstreamerInput. And > not because it's overengineered, IMHO, but because it's providing a lot of > functionality and solving a complicated set of problems. While I'd be > thrilled to have a SandboxInput replicating this functionality (mainly so > it could be used with Hindsight), I suspect that this is a much larger and > more difficult task than you might imagine. If you do end up going down > that road, I suggest you start with a tight, well-defined subset of > LogstreamerInput's features, and then work your way up from there. > > I can't speak to the issues that you're having with trying to use > `os.execute` or `io.popen` to run an external sleep command. It's very > possible that you're bumping up against Heka bugs with the behaviour you're > seeing. But that approach smells a bit off to me, I'm not surprised you're > having issues. You might consider one of the other suggested mechanisms > (see http://lua-users.org/wiki/SleepFunction), especially using > `socket.select`, or writing a trivial C extension that exposes a sleep > function. You also might try using Hindsight instead of Heka to test your > code, since that's a much lighter code base and is likely to have fewer (or > at least different) bugs. > > You might already have this on your radar, but if I were working on this > I'd almost certainly be using coroutines ( > http://lua-users.org/wiki/CoroutinesTutorial). > > Finally, you say that the reason you're not using LogstreamerInput is > because you need to use Lua's `io` module in your decoding code. If you're > okay with building your own custom Heka binary, and you don't plan on using > a SandboxManagerFilter to support dynamic sandbox injection (i.e. you trust > all of the Lua code you'll be deploying) then you might consider rebuilding > Heka with support for the io module allowed in all sandboxes. The code that > causes the `io` module to not be available is here: > > > https://github.com/mozilla-services/heka/blob/v0.10.0b1/sandbox/lua/lua_sandbox.go.in#L63 > > Hope this helps, > > -r > > > > On 11/18/2015 05:40 AM, Kipras Mancevičius wrote: > >> Hey guys, >> >> TL;DR any way to properly emulate LogStreamerInput behavior in Lua with >> a SandboxInput ? >> >> i'm having trouble trying to write a SandboxInput and since the docs >> aren't the most helpful regarding that (no examples) - i thought i will >> ask here. >> >> Any directions/pointers for making a SandboxInput that would >> continuously listen for new entries in a file? The reason why i'm not >> using a LogStreamerInput input with a SandboxDecoder is that i need to >> use the Lua 'io' module, and it's not available for SandboxDecoders only >> for SandboxInputs. >> >> The problems i'm having so far is: >> * if the input does not have a "ticker_interval" set and >> process_message() finishes - i get a "single run completed" entry in the >> log and the input never restarts >> >> * i tried using os.execute('sleep 1') / io.popen('sleep 1') after the >> input is exhausted to wait for a bit and then recheck if there is any >> new input, but: >> os.execute('sleep 1') works for sleeping and listening for new input, >> but stopping heka (Ctrl+C) does not work >> io.popen('sleep 1') also works for sleeping and listening for new input, >> but when trying to stop heka (Ctrl+C) on first Ctrl+C you get log >> entries "Shutdown initiated.", "Stop message sent to input >> '<your_input>'" but it does not shutdown (is there some way to handle >> the "stop message" in the SandboxInput??) and on some subsequent Ctrl+C >> it enters the infinte restart loop (read below) >> >> * if the input has a "ticker_interval" - it either also never restarts >> or restarts when you try to exit heka and enters a weird infinite loop >> state, where it looks be infinitely restarting, heka is at 100% cpu and >> you have to "kill -9" it to stop it. I think the infinite restart is >> caused by the input crashing for some reason and then my global input >> file position tracking is not updated and the input is restarted and so >> it is processing the same input again and again. This behavior depends >> on the set "ticker_interval", but i don't understand how those values >> work, e.g.: >> ticker_interval = 1000 : infinite restarting >> ticker_interval = 1000000000 : infinite restarting >> ticker_interval = 10000000000 : "single run completed" and never restarts >> ticker_interval = 100000000000 : infinite restarting >> This looks like some weird race conditions going on or smth.. >> >> I tried heka versions 0.9.2 and 0.10.0b1 and i'm getting the same >> results on both. >> >> Any way to properly emulate LogStreamerInput behavior in Lua with a >> SandboxInput ? >> >> >> >> _______________________________________________ >> Heka mailing list >> [email protected] >> https://mail.mozilla.org/listinfo/heka >> >> > -- Kipras Mancevičius | ZENEDGE | +37061814348 | [email protected] | www.zenedge.com This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.
_______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
