There are a few intentional changes between Heka and Hindsight. Looping messages in Heka has always been a bad idea so it was removed. There are a few API enhancements such as a protobuf stream reader and writer. Checkpoint are all managed by the Hindsight infrastructure (so much of the burden is removed from the plugin writer, this also alters the plugin API slightly). The write_message hack for Go has been removed since messages are immutable. read_config now has access to all related sandbox config options (standard and user defined). read_next_field is not supported (this will also be removed from Heka in 0.11).
In most cases you will find the Hindsight IOPS lower than Heka due to the much more efficient check pointing (btw Heka 0.11 is moving to a disk buffer everywhere). output_hi/input/* - contains the output from all of the input plugins output_hi/analysis/* - contains the output from all of the analysis plugins hindsight.cp - in the checkpoint file for all I/O (inputs, analysis, and output plugins) hindsight.tsv - in the self monitoring performance stats They files are all mandatory. They are the reason Hindsight has an at least once delivery guarantee and they provide valuable insight on system operation and performance. decode_message needs to be turned on for analysis and output plugins, I will enable it. Trink On Tue, Aug 11, 2015 at 1:54 AM, bruno binet <bruno.bi...@gmail.com> wrote: > I see, so I need to investigate how I can merge my multiple lua sandbox > filters into a single one. > > This make me wondering if there is some other differences between > Hindsight and Heka? > The fact that only one analysis plugin cannot consume the output of > another analysis plugin is the only difference beween Hindsight analysis > plugins and Heka filter sandbox plugins? > > Also I saw in another thread that Hindsight uses disk buffers at every > stage, so there's only ever one > message in memory at every step of the pipeline: does it mean Hindsight > will write much more frequently to the disk than Heka? This may be an issue > for me since we use a raspberry pi which disk is a sdcard or usb flash key. > > I see that some data is written to the output_path (output_hl/ directory > in my case): can you explain what are all these files: > $ tree output_hl/ > output_hl/ > |-- analysis > | `-- 0.log > |-- hindsight.cp > |-- hindsight.tsv > `-- input > `-- 0.log > > Can we avoid generating all these files? > > Last question: I don't manage to use the "read_next_field" or > "decode_message" api function from the output plugin, are they available? > > The following error is returned: > 1439280624615780495 [error] output_plugins terminated: > output/encode_metric.cfg msg: process_message() > _hl/output/encode_metric.lua:16: attempt to call global 'read_next_field' > (a nil value) > > or when I change my output plugin to use the decode_message api function: > 1439282566555139226 [error] output_plugins terminated: > output/encode_metric.cfg msg: process_message() > _hl/output/encode_metric.lua:15: attempt to call global 'decode_message' (a > nil value) > > > Thanks, > Bruno > > > On 10 August 2015 at 19:19, Michael Trinkala <mtrink...@mozilla.com> > wrote: > >> There is no message looping in Hindsight (one analysis plugin cannot >> consume the output of another analysis plugin). In your example the >> decoding should happen in the input. Heka has Inputs, splitters, and >> decoder (in Hindsight it is just an Input and common functionality can be >> split into modules for code reuse). This in general simplifies the >> configuration, is easier to follow (since everything is in one place) and >> has performance benefits as well. >> >> Trink >> >> On Mon, Aug 10, 2015 at 9:23 AM, bruno binet <bruno.bi...@gmail.com> >> wrote: >> >>> Back from vacations, I'm now playing again with Hindsight on a raspberry >>> pi. >>> As reported on github >>> https://github.com/trink/hindsight/issues/1#issuecomment-119593775 the >>> compilation now succeeds. >>> >>> So getting inspiration from the examples in the benchmarks directory, I >>> tried to create a Hindsight configuration to use my own lua sandboxes: I >>> can successfully read data from udp and use a filter to decode data, then I >>> would like to use another filter to handle generated messages, but I can't >>> get any message in the second filter. Does Hindsight support more than one >>> filter like Heka? >>> >>> Here is the Hindsight configuration, Lua sandboxes and output directory >>> generated by Hindsight: >>> https://github.com/bbinet/hindsight_hl_test >>> >>> Do you see anything wrong? Do I use hindsight correctly? >>> >>> Cheers, >>> Bruno >>> >>> On 8 July 2015 at 09:44, bruno binet <bruno.bi...@gmail.com> wrote: >>> >>>> Sure, I will try your branch and report possible new compilation issues >>>> in github. >>>> >>>> Cheers, >>>> Bruno >>>> >>>> On 7 July 2015 at 18:26, Michael Trinkala <mtrink...@mozilla.com> >>>> wrote: >>>> >>>>> I changed the checkpoint id to an unsigned long long. Can you test out >>>>> the branch and add any other compilation errors to the issue (closing out >>>>> this email thread). I am also taking suggestions/recommendations for a CI >>>>> build system that supports multiple platforms. TravisCI adds almost no >>>>> value since I am already building on a Debian based box. >>>>> >>>>> https://github.com/trink/hindsight/tree/issue_1 >>>>> >>>>> Thanks, >>>>> Trink >>>>> >>>>> On Tue, Jul 7, 2015 at 8:21 AM, bruno binet <bruno.bi...@gmail.com> >>>>> wrote: >>>>> >>>>>> Ok, thanks. >>>>>> And sorry, but I don't have a patch (don't know how to fix this kind >>>>>> of compilation issue). >>>>>> >>>>>> On 7 July 2015 at 16:17, Michael Trinkala <mtrink...@mozilla.com> >>>>>> wrote: >>>>>> >>>>>>> Yeah, I have only been building on Ubuntu and haven't done any cross >>>>>>> platform clean-up. Thanks for the build output I will fix those errors >>>>>>> (unless you already have a patch). >>>>>>> >>>>>>> Trink >>>>>>> >>>>>>> On Tue, Jul 7, 2015 at 5:57 AM, bruno binet <bruno.bi...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> I now have some time to do a few tests with Hindsight, so I tried >>>>>>>> to compile it on our targeted arm platform (raspberry pi), but I get >>>>>>>> the >>>>>>>> following error: >>>>>>>> >>>>>>>> root@hl-mc-9999-dev:~/hindsight/release# cmake >>>>>>>> -DCMAKE_BUILD_TYPE=release .. >>>>>>>> -- The C compiler identification is GNU 4.7.2 >>>>>>>> -- The CXX compiler identification is GNU 4.7.2 >>>>>>>> -- Check for working C compiler: /usr/bin/gcc >>>>>>>> -- Check for working C compiler: /usr/bin/gcc -- works >>>>>>>> -- Detecting C compiler ABI info >>>>>>>> -- Detecting C compiler ABI info - done >>>>>>>> -- Detecting C compile features >>>>>>>> -- Detecting C compile features - done >>>>>>>> -- Check for working CXX compiler: /usr/bin/g++ >>>>>>>> -- Check for working CXX compiler: /usr/bin/g++ -- works >>>>>>>> -- Detecting CXX compiler ABI info >>>>>>>> -- Detecting CXX compiler ABI info - done >>>>>>>> -- Detecting CXX compile features >>>>>>>> -- Detecting CXX compile features - done >>>>>>>> -- Found LUASANDBOX: /usr/local/lib/libluasandbox.so >>>>>>>> -- Configuring done >>>>>>>> -- Generating done >>>>>>>> -- Build files have been written to: /root/hindsight/release >>>>>>>> >>>>>>>> root@hl-mc-9999-dev:~/hindsight/release# make >>>>>>>> Scanning dependencies of target hindsight >>>>>>>> [ 2%] Building C object src/CMakeFiles/hindsight.dir/hindsight.c.o >>>>>>>> [ 4%] Building C object src/CMakeFiles/hindsight.dir/ >>>>>>>> hs_analysis_plugins.c.o >>>>>>>> [ 6%] Building C object src/CMakeFiles/hindsight.dir/ >>>>>>>> hs_checkpoint_reader.c.o >>>>>>>> /root/hindsight/src/hs_checkpoint_reader.c: In function >>>>>>>> 'find_first_id': >>>>>>>> /root/hindsight/src/hs_checkpoint_reader.c:46:3: error: large >>>>>>>> integer implicitly truncated to unsigned type [-Werror=overflow] >>>>>>>> /root/hindsight/src/hs_checkpoint_reader.c:55:3: error: comparison >>>>>>>> is always false due to limited range of data type [-Werror=type-limits] >>>>>>>> cc1: all warnings being treated as errors >>>>>>>> src/CMakeFiles/hindsight.dir/build.make:100: recipe for target >>>>>>>> 'src/CMakeFiles/hindsight.dir/hs_checkpoint_reader.c.o' failed >>>>>>>> make[2]: *** [src/CMakeFiles/hindsight.dir/hs_checkpoint_reader.c.o] >>>>>>>> Error 1 >>>>>>>> CMakeFiles/Makefile2:947: recipe for target >>>>>>>> 'src/CMakeFiles/hindsight.dir/all' failed >>>>>>>> make[1]: *** [src/CMakeFiles/hindsight.dir/all] Error 2 >>>>>>>> Makefile:146: recipe for target 'all' failed >>>>>>>> make: *** [all] Error 2 >>>>>>>> >>>>>>>> Do you know what is going on here? I guess this is an issue with >>>>>>>> the arm platform only? >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Bruno >>>>>>>> >>>>>>>> >>>>>>>> On 10 June 2015 at 18:41, bruno binet <bruno.bi...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Thanks a lot for your answers. >>>>>>>>> >>>>>>>>> And yes, I'm very interested in bootstrapping a first prototype of >>>>>>>>> my own data pipeline based on Hindsight so that I can compare the >>>>>>>>> performance on a raspberry pi. >>>>>>>>> (here is the current state of our Heka-based data pipeline: >>>>>>>>> https://bitbucket.org/helioslite/heka-hl-sandboxes) >>>>>>>>> So it would be great if you can give me the first instructions on >>>>>>>>> how to build and setup Hindsight. >>>>>>>>> >>>>>>>>> Thanks. >>>>>>>>> Bruno >>>>>>>>> >>>>>>>>> On 10 June 2015 at 18:18, Michael Trinkala <mtrink...@mozilla.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> - It is usable and being actively developed with the intent to >>>>>>>>>> move it into production later this year. >>>>>>>>>> - We are currently running production data through it for testing >>>>>>>>>> but it is not deployed in an official capacity. It has been very >>>>>>>>>> stable >>>>>>>>>> but until a more robust set of tests have been build out I will not >>>>>>>>>> consider it production ready. >>>>>>>>>> - Yes, it can decode/encode Heka protobuf format >>>>>>>>>> - Yes, the router/message matcher is complete. The only >>>>>>>>>> difference is that it supports Lua string pattern matching instead >>>>>>>>>> of re2 >>>>>>>>>> regexp (Heka 'Hostname =~ /^foo/' vs Hindsight 'Hostname =~ "^foo"') >>>>>>>>>> - Yes, but you would need a lua-socket input and output sandbox >>>>>>>>>> (see benchmarks/hsr_run for related examples) >>>>>>>>>> - No documentation yet, only examples in the benchmarks >>>>>>>>>> directory. I could have you bootstrapped in about 30 minutes (and >>>>>>>>>> hopefully turn that into a getting started guide) if you are >>>>>>>>>> interested. >>>>>>>>>> >>>>>>>>>> Implementation wise the only missing piece is support for >>>>>>>>>> dynamically loading plugins. The actual code to accomplish it is >>>>>>>>>> very >>>>>>>>>> small (just detecting files in the load directory and moving them to >>>>>>>>>> the >>>>>>>>>> run directory) but ideally it would be fronted by a web server and a >>>>>>>>>> GUI >>>>>>>>>> with access control and validation (a much larger effort and >>>>>>>>>> actually a >>>>>>>>>> separate project). >>>>>>>>>> >>>>>>>>>> Trink >>>>>>>>>> >>>>>>>>>> On Wed, Jun 10, 2015 at 8:15 AM, bruno binet < >>>>>>>>>> bruno.bi...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> I recently discovered the work pushed into the Hindsight >>>>>>>>>>> repository (https://github.com/trink/hindsight) which seems to >>>>>>>>>>> be a lightweight alternative to Heka, based on the lua sandbox. >>>>>>>>>>> The Hindsight vs Heka benchmarks are quite impressive. >>>>>>>>>>> >>>>>>>>>>> I'm currently running Heka on the raspberry pi (not so powerful) >>>>>>>>>>> device and the load average quickly increases and exceeds 1 when >>>>>>>>>>> Heka is >>>>>>>>>>> ingesting data, so Hindsight could be a good fit for us if it can >>>>>>>>>>> perform >>>>>>>>>>> better than Heka in terms of CPU cycles. >>>>>>>>>>> >>>>>>>>>>> What is the current status of Hindsight? Is it just an temporary >>>>>>>>>>> experiment or will it be maintained and actually used in production? >>>>>>>>>>> Is it currently usable and stable? >>>>>>>>>>> Is Hindsight able to decode and encode Heka protobuf format? >>>>>>>>>>> Does Hindsight have a complete router implementation to dispatch >>>>>>>>>>> messages to sandboxes like in Heka? >>>>>>>>>>> My use case is basically to read raw text data from UDP socket, >>>>>>>>>>> parse text data with lua patterns or lpeg, process data through a >>>>>>>>>>> few lua >>>>>>>>>>> sandbox filters, then write output messages both to a file >>>>>>>>>>> (protobuf heka >>>>>>>>>>> format) and a HTTP server (json format): can this be easily >>>>>>>>>>> accomplished >>>>>>>>>>> with Hindsight? >>>>>>>>>>> Is there any documentation somewhere to get started with >>>>>>>>>>> Hindsight? >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Bruno >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Heka mailing list >>>>>>>>>>> Heka@mozilla.org >>>>>>>>>>> https://mail.mozilla.org/listinfo/heka >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
_______________________________________________ Heka mailing list Heka@mozilla.org https://mail.mozilla.org/listinfo/heka
