Generally I only use the prune for reporting in which case I don't want it to clean up the results but it would be pretty easy to include the analysis directory as a config option.
Yes, there is up to a second of latency on detecting/reading the rolled logs. It is there to prevent rapid polling when the data flow is pretty low. When using a reasonably sized roll configuration the potential of up to a second of delay every few minutes hasn't been an issue for us. In fact most of the Hindsight backoffs/retries don't use anything smaller than a one second resolution as Hindsight is designed to be near real time (within a few seconds). Cool, the numbers are still looking good on this end too and it is pretty close to feature complete (with the recent addition of the dynamic loading). This issue is on my radar and will be addressed soon, as well as the documentation. Trink On Tue, Sep 15, 2015 at 8:04 AM, bruno binet <bruno.bi...@gmail.com> wrote: > Hi Michael, > > Thanks again for your answer. > I've also seen that you have checked in your "prune_input.lua plugin" on > github: > https://github.com/trink/hindsight/blob/master/sandboxes/input/prune_input.lua > This is very useful, and I'm wondering if you also have a similar pruning > plugin that could prune the output files produced by the analysis plugins? > > Also you said that we must be aware the roll transition can introduce up > to a second of latency in the analysis and output plugins: why does the > roll transition do introduce such a delay? Does this latency delay will be > decreased if we use your "prune_input.lua" plugin to remove the files that > would have been rolled? > > FYI, I've migrated my plugins and configuration from Heka to Hindsight, > and the performance is actually largely improved (CPU usage decreased by > ~10x, RAM usage decreased by ~80x !) > So thanks a lot for that great piece of software! > And that means that I can now use my Hindsight pipeline seamlessly on a > Raspberry PI 1 device without facing any performance issues anymore. :-) > > The last thing that I would like to fix before pushing Hindsight in > production is to be able to compile the master branch on Raspbery PI ARM > platform (for now I monkey patch the code for the compilation to succeed), > see github issue #7: https://github.com/trink/hindsight/pull/7 > > Cheers, > Bruno > > On 18 August 2015 at 06:18, Michael Trinkala <mtrink...@mozilla.com> > wrote: > >> Yes, I will get the plugins checked in shortly I just need to add a >> config option or two. >> >> run once - set the ticker_interval to 0 and return from process_message >> when you are done >> polling - set the ticker_interval > 0 and return from process_message, it >> will be called again when the ticker interval expires. >> continuous - don't return from process_message >> >> No, this feature does not need to be implemented in the infrastructure >> the plugin extends the functionality nicely and can be easily customized as >> needed. >> >> It totally depends on your throughput and message size, I don't expect >> any problems even if you roll frequently (be aware the roll transition can >> introduce up to a second of latency in the analysis and output plugins). >> >> Not at the moment since it is just two executables. >> >> Trink >> >> >> On Thu, Aug 13, 2015 at 5:36 AM, bruno binet <bruno.bi...@gmail.com> >> wrote: >> >>> >>> >>> On 12 August 2015 at 18:16, Michael Trinkala <mtrink...@mozilla.com> >>> wrote: >>> >>>> Most looping requirements can be flatten out: i.e., alerting can be >>>> handled in the output plugins in your example and >>>> aggregation/sessionization etc can be handled in the inputs. As for >>>> sharing: things fall into place when you start thinking about it from a >>>> module level instead of an individual plugin level. >>>> >>>> The locations are configurable 'output_path' so you can put the output >>>> files anywhere you want. >>>> >>> >>> Good, I will set it somewhere on the /tmp/ partition, which is already a >>> stored in RAM. >>> >>> >>>> I have some plugins (like a stdin, simple file, TCP. and a pruning >>>> (cleans up the output files when everyone in done with them) inputs; >>>> >>> >>> I'm interested by this pruning plugin: could you share it? Or don't you >>> think it would make sense for Hindsight to provide an option to >>> automaticallly clean up output files? >>> >>> >>>> heka protobuf and a payload outputs etc. but they haven't commited >>>> yet). As for ProcessInput the Input sandbox has access to os.execute so >>>> there won't be a generic version you can just call what you want and handle >>>> its output directly (Hindsight already supports run once, polling, and >>>> continuous input plugins) >>>> >>> >>> Can you explain briefly how to configure input plugins for these 3 >>> options (run once, polling, or continuous)? >>> >>> >>>> The output files will grow until 'output_size' (defaults to 64MiB) >>>> before they are rolled (they are not deleted by default). >>>> >>> >>> Ok. Would it make sense to introduce an option to delete them by default? >>> >>> >>>> I would not make it too small unless you need to prune really quickly >>>> generally I run with a config of 1GIB (on some of our systems that rolls >>>> several times a minute) space permitting I would just roll them after >>>> several minutes of what would be average data flow on your system) >>>> >>> >>> I plan to use a ram disk (tmpfs) to store these files (to avoid writing >>> too much on the sd card), so I would like them to stay small in order to >>> prevent eating too much RAM: does 5MB to 10MB seems reasonable to you? >>> >>> Also, do you plan to provide a debian package for Hindsight to ease >>> installation? >>> >>> Thanks for the great help! >>> Bruno >>> >>> >>>> >>>> Trink >>>> >>>> >>>> >>>> >>>> >>>> On Wed, Aug 12, 2015 at 1:55 AM, bruno binet <bruno.bi...@gmail.com> >>>> wrote: >>>> >>>>> >>>>> >>>>> On 12 August 2015 at 10:19, bruno binet <bruno.bi...@gmail.com> wrote: >>>>> >>>>>> Thanks for all this valuable information. >>>>>> >>>>>> On 11 August 2015 at 17:17, Michael Trinkala <mtrink...@mozilla.com> >>>>>> wrote: >>>>>> >>>>>>> There are a few intentional changes between Heka and Hindsight. >>>>>>> Looping messages in Heka has always been a bad idea so it was removed. >>>>>>> >>>>>> >>>>>> Personally I like the looping messages feature in Heka as it is very >>>>>> flexible and could be useful to share ready-to-use plugins. Also it >>>>>> supports processing messages through multiple ticker_interval which can >>>>>> be >>>>>> useful (alerting, aggregations). >>>>>> >>>>>> >>>>>>> There are a few API enhancements such as a protobuf stream reader >>>>>>> and writer. Checkpoint are all managed by the Hindsight infrastructure >>>>>>> (so >>>>>>> much of the burden is removed from the plugin writer, this also alters >>>>>>> the >>>>>>> plugin API slightly). The write_message hack for Go has been removed >>>>>>> since >>>>>>> messages are immutable. read_config now has access to all related >>>>>>> sandbox >>>>>>> config options (standard and user defined). read_next_field is not >>>>>>> supported (this will also be removed from Heka in 0.11). >>>>>>> >>>>>>> In most cases you will find the Hindsight IOPS lower than Heka due >>>>>>> to the much more efficient check pointing (btw Heka 0.11 is moving to a >>>>>>> disk buffer everywhere). >>>>>>> >>>>>> >>>>>> Great, that is good to know. >>>>>> >>>>>> >>>>>>> output_hi/input/* - contains the output from all of the input plugins >>>>>>> output_hi/analysis/* - contains the output from all of the analysis >>>>>>> plugins >>>>>>> >>>>>> >>>>> Are the above files always growing? >>>>> I suppose the output_limit configuration allow us limit their size: >>>>> what are the implications if I limit their size to a few KB? Will it >>>>> reduce >>>>> Hindsight performance? >>>>> >>>>> >>>>>> hindsight.cp - in the checkpoint file for all I/O (inputs, analysis, >>>>>>> and output plugins) >>>>>>> hindsight.tsv - in the self monitoring performance stats >>>>>>> >>>>>>> They files are all mandatory. They are the reason Hindsight has an >>>>>>> at least once delivery guarantee and they provide valuable insight on >>>>>>> system operation and performance. >>>>>>> >>>>>> >>>>>> If I don't need delivery guarantee, do you think it could make sense >>>>>> to move these files to a ramdisk (tmpfs) partition in order to preserve >>>>>> the >>>>>> flash sd/usb card? >>>>>> >>>>>> >>>>>>> decode_message needs to be turned on for analysis and output >>>>>>> plugins, I will enable it. >>>>>>> >>>>>> >>>>>> Ok, thank you: this is now working as expected. >>>>>> >>>>>> Also, do you plan to implement some additional lua modules to help >>>>>> build input sandboxes similar to Heka input plugins (like the >>>>>> FilePollingInput, the ProcessInput, or the LogstreamerInput)? >>>>>> >>>>>> >>>>>>> >>>>>>> Trink >>>>>>> >>>>>>> >>>>>>> On Tue, Aug 11, 2015 at 1:54 AM, bruno binet <bruno.bi...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> I see, so I need to investigate how I can merge my multiple lua >>>>>>>> sandbox filters into a single one. >>>>>>>> >>>>>>>> This make me wondering if there is some other differences between >>>>>>>> Hindsight and Heka? >>>>>>>> The fact that only one analysis plugin cannot consume the output of >>>>>>>> another analysis plugin is the only difference beween Hindsight >>>>>>>> analysis >>>>>>>> plugins and Heka filter sandbox plugins? >>>>>>>> >>>>>>>> Also I saw in another thread that Hindsight uses disk buffers at >>>>>>>> every stage, so there's only ever one >>>>>>>> message in memory at every step of the pipeline: does it mean >>>>>>>> Hindsight will write much more frequently to the disk than Heka? This >>>>>>>> may >>>>>>>> be an issue for me since we use a raspberry pi which disk is a sdcard >>>>>>>> or >>>>>>>> usb flash key. >>>>>>>> >>>>>>>> I see that some data is written to the output_path (output_hl/ >>>>>>>> directory in my case): can you explain what are all these files: >>>>>>>> $ tree output_hl/ >>>>>>>> output_hl/ >>>>>>>> |-- analysis >>>>>>>> | `-- 0.log >>>>>>>> |-- hindsight.cp >>>>>>>> |-- hindsight.tsv >>>>>>>> `-- input >>>>>>>> `-- 0.log >>>>>>>> >>>>>>>> Can we avoid generating all these files? >>>>>>>> >>>>>>>> Last question: I don't manage to use the "read_next_field" or >>>>>>>> "decode_message" api function from the output plugin, are they >>>>>>>> available? >>>>>>>> >>>>>>>> The following error is returned: >>>>>>>> 1439280624615780495 [error] output_plugins terminated: >>>>>>>> output/encode_metric.cfg msg: process_message() >>>>>>>> _hl/output/encode_metric.lua:16: attempt to call global >>>>>>>> 'read_next_field' >>>>>>>> (a nil value) >>>>>>>> >>>>>>>> or when I change my output plugin to use the decode_message api >>>>>>>> function: >>>>>>>> 1439282566555139226 [error] output_plugins terminated: >>>>>>>> output/encode_metric.cfg msg: process_message() >>>>>>>> _hl/output/encode_metric.lua:15: attempt to call global >>>>>>>> 'decode_message' (a >>>>>>>> nil value) >>>>>>>> >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Bruno >>>>>>>> >>>>>>>> >>>>>>>> On 10 August 2015 at 19:19, Michael Trinkala <mtrink...@mozilla.com >>>>>>>> > wrote: >>>>>>>> >>>>>>>>> There is no message looping in Hindsight (one analysis plugin >>>>>>>>> cannot consume the output of another analysis plugin). In your >>>>>>>>> example the >>>>>>>>> decoding should happen in the input. Heka has Inputs, splitters, and >>>>>>>>> decoder (in Hindsight it is just an Input and common functionality >>>>>>>>> can be >>>>>>>>> split into modules for code reuse). This in general simplifies the >>>>>>>>> configuration, is easier to follow (since everything is in one place) >>>>>>>>> and >>>>>>>>> has performance benefits as well. >>>>>>>>> >>>>>>>>> Trink >>>>>>>>> >>>>>>>>> On Mon, Aug 10, 2015 at 9:23 AM, bruno binet < >>>>>>>>> bruno.bi...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Back from vacations, I'm now playing again with Hindsight on a >>>>>>>>>> raspberry pi. >>>>>>>>>> As reported on github >>>>>>>>>> https://github.com/trink/hindsight/issues/1#issuecomment-119593775 >>>>>>>>>> the compilation now succeeds. >>>>>>>>>> >>>>>>>>>> So getting inspiration from the examples in the benchmarks >>>>>>>>>> directory, I tried to create a Hindsight configuration to use my own >>>>>>>>>> lua >>>>>>>>>> sandboxes: I can successfully read data from udp and use a filter to >>>>>>>>>> decode >>>>>>>>>> data, then I would like to use another filter to handle generated >>>>>>>>>> messages, >>>>>>>>>> but I can't get any message in the second filter. Does Hindsight >>>>>>>>>> support >>>>>>>>>> more than one filter like Heka? >>>>>>>>>> >>>>>>>>>> Here is the Hindsight configuration, Lua sandboxes and output >>>>>>>>>> directory generated by Hindsight: >>>>>>>>>> https://github.com/bbinet/hindsight_hl_test >>>>>>>>>> >>>>>>>>>> Do you see anything wrong? Do I use hindsight correctly? >>>>>>>>>> >>>>>>>>>> Cheers, >>>>>>>>>> Bruno >>>>>>>>>> >>>>>>>>>> On 8 July 2015 at 09:44, bruno binet <bruno.bi...@gmail.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Sure, I will try your branch and report possible new compilation >>>>>>>>>>> issues in github. >>>>>>>>>>> >>>>>>>>>>> Cheers, >>>>>>>>>>> Bruno >>>>>>>>>>> >>>>>>>>>>> On 7 July 2015 at 18:26, Michael Trinkala <mtrink...@mozilla.com >>>>>>>>>>> > wrote: >>>>>>>>>>> >>>>>>>>>>>> I changed the checkpoint id to an unsigned long long. Can you >>>>>>>>>>>> test out the branch and add any other compilation errors to the >>>>>>>>>>>> issue >>>>>>>>>>>> (closing out this email thread). I am also taking >>>>>>>>>>>> suggestions/recommendations for a CI build system that supports >>>>>>>>>>>> multiple >>>>>>>>>>>> platforms. TravisCI adds almost no value since I am already >>>>>>>>>>>> building on a >>>>>>>>>>>> Debian based box. >>>>>>>>>>>> >>>>>>>>>>>> https://github.com/trink/hindsight/tree/issue_1 >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> Trink >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Jul 7, 2015 at 8:21 AM, bruno binet < >>>>>>>>>>>> bruno.bi...@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Ok, thanks. >>>>>>>>>>>>> And sorry, but I don't have a patch (don't know how to fix >>>>>>>>>>>>> this kind of compilation issue). >>>>>>>>>>>>> >>>>>>>>>>>>> On 7 July 2015 at 16:17, Michael Trinkala < >>>>>>>>>>>>> mtrink...@mozilla.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Yeah, I have only been building on Ubuntu and haven't done >>>>>>>>>>>>>> any cross platform clean-up. Thanks for the build output I will >>>>>>>>>>>>>> fix those >>>>>>>>>>>>>> errors (unless you already have a patch). >>>>>>>>>>>>>> >>>>>>>>>>>>>> Trink >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Tue, Jul 7, 2015 at 5:57 AM, bruno binet < >>>>>>>>>>>>>> bruno.bi...@gmail.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> I now have some time to do a few tests with Hindsight, so I >>>>>>>>>>>>>>> tried to compile it on our targeted arm platform (raspberry >>>>>>>>>>>>>>> pi), but I get >>>>>>>>>>>>>>> the following error: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> root@hl-mc-9999-dev:~/hindsight/release# cmake >>>>>>>>>>>>>>> -DCMAKE_BUILD_TYPE=release .. >>>>>>>>>>>>>>> -- The C compiler identification is GNU 4.7.2 >>>>>>>>>>>>>>> -- The CXX compiler identification is GNU 4.7.2 >>>>>>>>>>>>>>> -- Check for working C compiler: /usr/bin/gcc >>>>>>>>>>>>>>> -- Check for working C compiler: /usr/bin/gcc -- works >>>>>>>>>>>>>>> -- Detecting C compiler ABI info >>>>>>>>>>>>>>> -- Detecting C compiler ABI info - done >>>>>>>>>>>>>>> -- Detecting C compile features >>>>>>>>>>>>>>> -- Detecting C compile features - done >>>>>>>>>>>>>>> -- Check for working CXX compiler: /usr/bin/g++ >>>>>>>>>>>>>>> -- Check for working CXX compiler: /usr/bin/g++ -- works >>>>>>>>>>>>>>> -- Detecting CXX compiler ABI info >>>>>>>>>>>>>>> -- Detecting CXX compiler ABI info - done >>>>>>>>>>>>>>> -- Detecting CXX compile features >>>>>>>>>>>>>>> -- Detecting CXX compile features - done >>>>>>>>>>>>>>> -- Found LUASANDBOX: /usr/local/lib/libluasandbox.so >>>>>>>>>>>>>>> -- Configuring done >>>>>>>>>>>>>>> -- Generating done >>>>>>>>>>>>>>> -- Build files have been written to: /root/hindsight/release >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> root@hl-mc-9999-dev:~/hindsight/release# make >>>>>>>>>>>>>>> Scanning dependencies of target hindsight >>>>>>>>>>>>>>> [ 2%] Building C object src/CMakeFiles/hindsight.dir/ >>>>>>>>>>>>>>> hindsight.c.o >>>>>>>>>>>>>>> [ 4%] Building C object src/CMakeFiles/hindsight.dir/ >>>>>>>>>>>>>>> hs_analysis_plugins.c.o >>>>>>>>>>>>>>> [ 6%] Building C object src/CMakeFiles/hindsight.dir/ >>>>>>>>>>>>>>> hs_checkpoint_reader.c.o >>>>>>>>>>>>>>> /root/hindsight/src/hs_checkpoint_reader.c: In function >>>>>>>>>>>>>>> 'find_first_id': >>>>>>>>>>>>>>> /root/hindsight/src/hs_checkpoint_reader.c:46:3: error: >>>>>>>>>>>>>>> large integer implicitly truncated to unsigned type >>>>>>>>>>>>>>> [-Werror=overflow] >>>>>>>>>>>>>>> /root/hindsight/src/hs_checkpoint_reader.c:55:3: error: >>>>>>>>>>>>>>> comparison is always false due to limited range of data type >>>>>>>>>>>>>>> [-Werror=type-limits] >>>>>>>>>>>>>>> cc1: all warnings being treated as errors >>>>>>>>>>>>>>> src/CMakeFiles/hindsight.dir/build.make:100: recipe for >>>>>>>>>>>>>>> target 'src/CMakeFiles/hindsight.dir/hs_checkpoint_reader.c.o' >>>>>>>>>>>>>>> failed >>>>>>>>>>>>>>> make[2]: *** >>>>>>>>>>>>>>> [src/CMakeFiles/hindsight.dir/hs_checkpoint_reader.c.o] >>>>>>>>>>>>>>> Error 1 >>>>>>>>>>>>>>> CMakeFiles/Makefile2:947: recipe for target >>>>>>>>>>>>>>> 'src/CMakeFiles/hindsight.dir/all' failed >>>>>>>>>>>>>>> make[1]: *** [src/CMakeFiles/hindsight.dir/all] Error 2 >>>>>>>>>>>>>>> Makefile:146: recipe for target 'all' failed >>>>>>>>>>>>>>> make: *** [all] Error 2 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Do you know what is going on here? I guess this is an issue >>>>>>>>>>>>>>> with the arm platform only? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Cheers, >>>>>>>>>>>>>>> Bruno >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 10 June 2015 at 18:41, bruno binet <bruno.bi...@gmail.com >>>>>>>>>>>>>>> > wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks a lot for your answers. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> And yes, I'm very interested in bootstrapping a first >>>>>>>>>>>>>>>> prototype of my own data pipeline based on Hindsight so that I >>>>>>>>>>>>>>>> can compare >>>>>>>>>>>>>>>> the performance on a raspberry pi. >>>>>>>>>>>>>>>> (here is the current state of our Heka-based data pipeline: >>>>>>>>>>>>>>>> https://bitbucket.org/helioslite/heka-hl-sandboxes) >>>>>>>>>>>>>>>> So it would be great if you can give me the first >>>>>>>>>>>>>>>> instructions on how to build and setup Hindsight. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks. >>>>>>>>>>>>>>>> Bruno >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 10 June 2015 at 18:18, Michael Trinkala < >>>>>>>>>>>>>>>> mtrink...@mozilla.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> - It is usable and being actively developed with the >>>>>>>>>>>>>>>>> intent to move it into production later this year. >>>>>>>>>>>>>>>>> - We are currently running production data through it for >>>>>>>>>>>>>>>>> testing but it is not deployed in an official capacity. It >>>>>>>>>>>>>>>>> has been very >>>>>>>>>>>>>>>>> stable but until a more robust set of tests have been build >>>>>>>>>>>>>>>>> out I will not >>>>>>>>>>>>>>>>> consider it production ready. >>>>>>>>>>>>>>>>> - Yes, it can decode/encode Heka protobuf format >>>>>>>>>>>>>>>>> - Yes, the router/message matcher is complete. The only >>>>>>>>>>>>>>>>> difference is that it supports Lua string pattern matching >>>>>>>>>>>>>>>>> instead of re2 >>>>>>>>>>>>>>>>> regexp (Heka 'Hostname =~ /^foo/' vs Hindsight 'Hostname =~ >>>>>>>>>>>>>>>>> "^foo"') >>>>>>>>>>>>>>>>> - Yes, but you would need a lua-socket input and output >>>>>>>>>>>>>>>>> sandbox (see benchmarks/hsr_run for related examples) >>>>>>>>>>>>>>>>> - No documentation yet, only examples in the benchmarks >>>>>>>>>>>>>>>>> directory. I could have you bootstrapped in about 30 minutes >>>>>>>>>>>>>>>>> (and >>>>>>>>>>>>>>>>> hopefully turn that into a getting started guide) if you are >>>>>>>>>>>>>>>>> interested. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Implementation wise the only missing piece is support for >>>>>>>>>>>>>>>>> dynamically loading plugins. The actual code to accomplish >>>>>>>>>>>>>>>>> it is very >>>>>>>>>>>>>>>>> small (just detecting files in the load directory and moving >>>>>>>>>>>>>>>>> them to the >>>>>>>>>>>>>>>>> run directory) but ideally it would be fronted by a web >>>>>>>>>>>>>>>>> server and a GUI >>>>>>>>>>>>>>>>> with access control and validation (a much larger effort and >>>>>>>>>>>>>>>>> actually a >>>>>>>>>>>>>>>>> separate project). >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Trink >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Wed, Jun 10, 2015 at 8:15 AM, bruno binet < >>>>>>>>>>>>>>>>> bruno.bi...@gmail.com> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I recently discovered the work pushed into the Hindsight >>>>>>>>>>>>>>>>>> repository (https://github.com/trink/hindsight) which >>>>>>>>>>>>>>>>>> seems to be a lightweight alternative to Heka, based on the >>>>>>>>>>>>>>>>>> lua sandbox. >>>>>>>>>>>>>>>>>> The Hindsight vs Heka benchmarks are quite impressive. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I'm currently running Heka on the raspberry pi (not so >>>>>>>>>>>>>>>>>> powerful) device and the load average quickly increases and >>>>>>>>>>>>>>>>>> exceeds 1 when >>>>>>>>>>>>>>>>>> Heka is ingesting data, so Hindsight could be a good fit for >>>>>>>>>>>>>>>>>> us if it can >>>>>>>>>>>>>>>>>> perform better than Heka in terms of CPU cycles. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> What is the current status of Hindsight? Is it just an >>>>>>>>>>>>>>>>>> temporary experiment or will it be maintained and actually >>>>>>>>>>>>>>>>>> used in >>>>>>>>>>>>>>>>>> production? >>>>>>>>>>>>>>>>>> Is it currently usable and stable? >>>>>>>>>>>>>>>>>> Is Hindsight able to decode and encode Heka protobuf >>>>>>>>>>>>>>>>>> format? >>>>>>>>>>>>>>>>>> Does Hindsight have a complete router implementation to >>>>>>>>>>>>>>>>>> dispatch messages to sandboxes like in Heka? >>>>>>>>>>>>>>>>>> My use case is basically to read raw text data from UDP >>>>>>>>>>>>>>>>>> socket, parse text data with lua patterns or lpeg, process >>>>>>>>>>>>>>>>>> data through a >>>>>>>>>>>>>>>>>> few lua sandbox filters, then write output messages both to >>>>>>>>>>>>>>>>>> a file >>>>>>>>>>>>>>>>>> (protobuf heka format) and a HTTP server (json format): can >>>>>>>>>>>>>>>>>> this be easily >>>>>>>>>>>>>>>>>> accomplished with Hindsight? >>>>>>>>>>>>>>>>>> Is there any documentation somewhere to get started with >>>>>>>>>>>>>>>>>> Hindsight? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>> Bruno >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>> Heka mailing list >>>>>>>>>>>>>>>>>> Heka@mozilla.org >>>>>>>>>>>>>>>>>> https://mail.mozilla.org/listinfo/heka >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
_______________________________________________ Heka mailing list Heka@mozilla.org https://mail.mozilla.org/listinfo/heka
