Coroutines can be enabled (3 lines above the link Rob sent, same type of modification as the io tweak)
Trink On Thu, Nov 19, 2015 at 1:50 AM, Kipras Mancevičius <kip...@zenedge.com> wrote: > Hi, thanks for all the info. I'll look into the options again. Most > probably going to recompile Heka and enable the "io" module. > > As for coroutines - i did try to use them as they seemed to be the way to > go for implementing this, however they didn't work - it seems that > "yield()' is not supported in the Heka Lua sandbox (it complains about an > undefined global). > > On Thu, Nov 19, 2015 at 12:16 AM, Rob Miller <rmil...@mozilla.com> wrote: > >> Hrm. There's a *lot* of code and complexity in the LogstreamerInput. And >> not because it's overengineered, IMHO, but because it's providing a lot of >> functionality and solving a complicated set of problems. While I'd be >> thrilled to have a SandboxInput replicating this functionality (mainly so >> it could be used with Hindsight), I suspect that this is a much larger and >> more difficult task than you might imagine. If you do end up going down >> that road, I suggest you start with a tight, well-defined subset of >> LogstreamerInput's features, and then work your way up from there. >> >> I can't speak to the issues that you're having with trying to use >> `os.execute` or `io.popen` to run an external sleep command. It's very >> possible that you're bumping up against Heka bugs with the behaviour you're >> seeing. But that approach smells a bit off to me, I'm not surprised you're >> having issues. You might consider one of the other suggested mechanisms >> (see http://lua-users.org/wiki/SleepFunction), especially using >> `socket.select`, or writing a trivial C extension that exposes a sleep >> function. You also might try using Hindsight instead of Heka to test your >> code, since that's a much lighter code base and is likely to have fewer (or >> at least different) bugs. >> >> You might already have this on your radar, but if I were working on this >> I'd almost certainly be using coroutines ( >> http://lua-users.org/wiki/CoroutinesTutorial). >> >> Finally, you say that the reason you're not using LogstreamerInput is >> because you need to use Lua's `io` module in your decoding code. If you're >> okay with building your own custom Heka binary, and you don't plan on using >> a SandboxManagerFilter to support dynamic sandbox injection (i.e. you trust >> all of the Lua code you'll be deploying) then you might consider rebuilding >> Heka with support for the io module allowed in all sandboxes. The code that >> causes the `io` module to not be available is here: >> >> >> https://github.com/mozilla-services/heka/blob/v0.10.0b1/sandbox/lua/lua_sandbox.go.in#L63 >> >> Hope this helps, >> >> -r >> >> >> >> On 11/18/2015 05:40 AM, Kipras Mancevičius wrote: >> >>> Hey guys, >>> >>> TL;DR any way to properly emulate LogStreamerInput behavior in Lua with >>> a SandboxInput ? >>> >>> i'm having trouble trying to write a SandboxInput and since the docs >>> aren't the most helpful regarding that (no examples) - i thought i will >>> ask here. >>> >>> Any directions/pointers for making a SandboxInput that would >>> continuously listen for new entries in a file? The reason why i'm not >>> using a LogStreamerInput input with a SandboxDecoder is that i need to >>> use the Lua 'io' module, and it's not available for SandboxDecoders only >>> for SandboxInputs. >>> >>> The problems i'm having so far is: >>> * if the input does not have a "ticker_interval" set and >>> process_message() finishes - i get a "single run completed" entry in the >>> log and the input never restarts >>> >>> * i tried using os.execute('sleep 1') / io.popen('sleep 1') after the >>> input is exhausted to wait for a bit and then recheck if there is any >>> new input, but: >>> os.execute('sleep 1') works for sleeping and listening for new input, >>> but stopping heka (Ctrl+C) does not work >>> io.popen('sleep 1') also works for sleeping and listening for new input, >>> but when trying to stop heka (Ctrl+C) on first Ctrl+C you get log >>> entries "Shutdown initiated.", "Stop message sent to input >>> '<your_input>'" but it does not shutdown (is there some way to handle >>> the "stop message" in the SandboxInput??) and on some subsequent Ctrl+C >>> it enters the infinte restart loop (read below) >>> >>> * if the input has a "ticker_interval" - it either also never restarts >>> or restarts when you try to exit heka and enters a weird infinite loop >>> state, where it looks be infinitely restarting, heka is at 100% cpu and >>> you have to "kill -9" it to stop it. I think the infinite restart is >>> caused by the input crashing for some reason and then my global input >>> file position tracking is not updated and the input is restarted and so >>> it is processing the same input again and again. This behavior depends >>> on the set "ticker_interval", but i don't understand how those values >>> work, e.g.: >>> ticker_interval = 1000 : infinite restarting >>> ticker_interval = 1000000000 : infinite restarting >>> ticker_interval = 10000000000 : "single run completed" and never restarts >>> ticker_interval = 100000000000 : infinite restarting >>> This looks like some weird race conditions going on or smth.. >>> >>> I tried heka versions 0.9.2 and 0.10.0b1 and i'm getting the same >>> results on both. >>> >>> Any way to properly emulate LogStreamerInput behavior in Lua with a >>> SandboxInput ? >>> >>> >>> >>> _______________________________________________ >>> Heka mailing list >>> Heka@mozilla.org >>> https://mail.mozilla.org/listinfo/heka >>> >>> >> > > > -- > Kipras Mancevičius | ZENEDGE | +37061814348 | kip...@zenedge.com | > www.zenedge.com > > This email and any files transmitted with it are confidential and intended > solely for the use of the individual or entity to whom they are addressed. > If you have received this email in error please notify the system manager. > This message contains confidential information and is intended only for the > individual named. If you are not the named addressee you should not > disseminate, distribute or copy this e-mail. Please notify the sender > immediately by e-mail if you have received this e-mail by mistake and > delete this e-mail from your system. If you are not the intended recipient > you are notified that disclosing, copying, distributing or taking any > action in reliance on the contents of this information is strictly > prohibited. > > > _______________________________________________ > Heka mailing list > Heka@mozilla.org > https://mail.mozilla.org/listinfo/heka > >
_______________________________________________ Heka mailing list Heka@mozilla.org https://mail.mozilla.org/listinfo/heka
