Use the output and inject_message functions. i.e.
output(my_table)
inject_message()
-- you can optionally add a type/title for easier downstream
filtering/consumption especially if the decoder can output multiple json
schemas.
-- inject_message("json", "my_schema1")
Trink
- Original
I threw together a simple on-line LPeg grammar tester for those that are
interested.
http://lpeg.trink.com
Trink
___
Heka mailing list
Heka@mozilla.org
https://mail.mozilla.org/listinfo/heka
See https://github.com/mozilla-services/heka/issues/407
Thanks,
Trink
- Original Message -
> From: "Kushal Pisavadia"
> To: Heka@mozilla.org
> Sent: Friday, January 3, 2014 12:56:12 AM
> Subject: [heka] Config: using multiple files
> Heka currently expects a single config file to be
The LogfileInput is not limited to traditional lines. The delimiter can occur
anywhere in the input defining what constitutes a line but you cannot use the
regex end of line anchor for it to work. Based on your example it looks likes
you are dealing with traditional log lines and batching them i
Adding shared library modules is not supported in the sandbox/Heka. You would
have to modify the sandbox code/build and update the Heka externals to use that
version.
Trink
- Original Message -
> From: "Mark Melling"
> To: heka@mozilla.org
> Sent: Wednesday, January 8, 2014 3:54:07 A
ctest -R graphite
- Original Message -
> From: "Lev Brouk"
> To: heka@mozilla.org
> Sent: Thursday, January 30, 2014 4:51:58 PM
> Subject: [heka] running tests for a single (go) plugin -- how?
> Hi,
> I am adding some functionality and tests for graphite and nagios plugins.
> The onl
We recommend the latest dev branch. It should always be reasonably stable (we
do push the new features out to some of our production systems); and obviously
integration will be a lot easier since you will be using the latest APIs.
Trink
- Original Message -
> From: "David Birdsong"
This should work fine (it is the same structure as goprotobuf)
hg_clone(https://code.google.com/p/snappy-go )
You would have to import it as
import "code.google.com/p/snappy-go/snappy"
Trink
- Original Message -
> From: "David Birdsong"
> To: "heka"
> Sent: Friday, January 31, 2
import "code.google.com/p/goprotobuf/proto"
import "github.com/mozilla-services/heka/message"
b := // []byte value from your code
msg := new(message.Message)
err := proto.Unmarshal(b, msg)
Trink
- Original Message -
> From: "David Birdsong"
> To: "heka"
> Sent: Friday, January 31
For use in an input plugin 'b' should be written into
pipeline.PipelinePack.MsgBytes and the pack should be sent to a ProtobufDecoder
(which will perform the de-serialization below)
Trink
- Original Message -
> From: "David Birdsong"
> To: "Michael
Thanks for your interest in Heka. The current (v0.5) TcpOutput has disk queuing
and should be able to support those needs. Note: the connection retry does have
a timed back-off so it may not retry immediately when a new connection is
established (the delay could be up to 10 seconds).
Thanks,
I don't believe anyone has tried.
There are a couple of ways you can specify the compiler:
- in the toolchain file set this variable CMAKE_C_COMPILER (looks like you have
a typo in your file)
- in the environment set the CC environment variable and cmake will pick it up
during generation
In
If you aren't on Linux, yes, Otherwise, just some unit test output comparisons
will have to be cleaned up.
Trink
- Original Message -
> From: "David Birdsong"
> To: "heka"
> Sent: Thursday, May 22, 2014 9:07:37 AM
> Subject: [heka] here be lions
> Is there something bad waiting to
Hi,
Just wanted to give everyone a heads up that a breaking change to the Lua
sandbox will be landing on the dev branch of Heka soon.
Summary:
1) inject_message is no longer overloaded. Only this signature is valid:
https://hekad.readthedocs.org/en/latest/sandbox/index.html#inject-message-
Not explicitly, the value has to be compared against something. For a string
field something like this would work Fields[t] =~ /.?/
Trink
- Original Message -
> From: "briche arnaud"
> To: heka@mozilla.org
> Sent: Thursday, June 19, 2014 11:19:46 AM
> Subject: [heka] Test field exist
Hi all,
More updates on breaking changes... I'm about to merged a PR into
the dev branch removing the PayloadJsonDecoder. It has been deprecated since
the v0.5.0 release with the introduction of cjson module in the Lua sandbox.
If you need some help migrating off of it, please ask here or find us
Correct optional fields can be omitted from the Heka message.
The uuid and timestamp should remain required and set by the message creator,
otherwise:
- If the message creator was delayed in sending the messages to Heka when the
backlog was processed all messages would basically be given the s
The functionality has been merged into the dev branch.
https://github.com/mozilla-services/heka/issues/958
Trink
- Original Message -
> From: "Michael Trinkala"
> To: "briche arnaud"
> Cc: heka@mozilla.org
> Sent: Thursday, June 19, 2014 1:12:34 PM
It is just a little C++ application I did over the Christmas holiday therefore
a quick review and some updates to the readme would be useful before general
release. I will try to get it published within the next few days. Here is
what you will need to build it.
Prerequisites {#mainpage}
Just added a config file and shell script to make running it a little easier,
let me know if you have any issues.
https://github.com/trink/lpeg_tester
Trink
- Original Message -
> From: "Michael Trinkala"
> To: "Sebastian Kurfürst"
> Cc: heka@mozilla.o
1) The decoder has several issues.
- replace 'null' with 'nil'
- if a '@timestamp' field doesn't exist (or is not a string) in the JSON the
decoder will terminate
- inject_message should be wrapped in a pcall (since you are converting an
untrusted json structure)
- if you are using Heka 0.6+ you
Set ticker_interval to 30
* ticker_interval (uint):
The number of seconds to wait between each run of command . Defaults to 15. A
ticker_interval of 0 indicates that the command is run only once, useful for
long running processes.
Retries applies to the plugin itself and not the command
If there is enough interest I could work out of the SF office that day.
For those of you not in the bay area I am also open to doing something online.
e.g., Etherpad and the grammar tester has worked quite well for LPeg training
sessions and small tasks like sandbox decoder/filter development a
It is not possible to override the maximum message size in the toml
configuration. At the moment the only way to change it is update the constant
in message.go and recompile.
Trink
- Original Message -
> From: "Sebastian YEPES FERNANDEZ"
> To: heka@mozilla.org
> Sent: Thursday, Octo
Yes for any Lua based modules they all have to reside under the same directory
per plugin or sandbox manager (and only contain alphanumeric or hyphen
characters). When debugging I either put the module I am developing in the
lua_module directory or symlink the modules I need from that directory
Heka Jam
06:30 - 11:00 PST (online: parsing, transformation, analysis, code review) We
can do a combination of IRC, etherpad, and the LPeg grammar tester.
11:00 - 13:00 PST (out for lunch/in-transit)
13:00 - 17:00 PST (online/on-site: parsing, transformation, analysis, code
review) Mozil
You would actually use a SandboxDecoder for this task. The plugin wouldn't
consist of much more than this grammar: http://lpeg.trink.com/share/nvlist
It is also the type of thing we will be discussing at the 'Heka Jam' tomorrow.
https://mail.mozilla.org/pipermail/heka/2014-November/000251.html
https://mail.mozilla.org/pipermail/heka/2014-November/000261.html (will be
online too)
See http://lpeg.trink.com/share/csv for an example of handling quoted strings.
Trink
- Original Message -
> From: "Greg Cope"
> To: "Michael Trinkala"
> Cc:
We usually throttle them at the source like
https://github.com/mozilla-services/heka/blob/dev/sandbox/lua/filters/http_status.lua#L85
but there is nothing to prevent you from adding global throttling or
aggregation in the alert encoder (by host, plugin, application... whatever)
Trink
-
You can strip it in a SandboxDecoder (on the way into Heka) or in a
SandboxEncoder (on the way out) depending on your requirements. You would
just do a search and replace on the request_uri (if you are using the
common log format grammars).
On another note, having sensitive information in the URI
Our sandbox unit tests can be found here:
https://github.com/mozilla-services/heka/tree/dev/sandbox/plugins They
should provide a reasonable example of how it can be done.
Trink
On Thu, Jan 22, 2015 at 10:42 AM, Djamel F. wrote:
> I wrote many sandboxes to treat data, and I would
> to
> do s
The log_format configuration does not match the sample output you provided
(beginning at $request_time)
On Thu, Jan 22, 2015 at 12:43 AM, Yanhong Wu wrote:
> sorry, the above $cookie_username was broken when being copied from two
> lines. The real variable name is $cookie_username, no blank with
Oops, didn't respond to all yesterday...
-- Forwarded message --
From: Michael Trinkala
Date: Sat, Jan 31, 2015 at 2:16 PM
Subject: Re: [heka] Help with lpeg grammar for KV-tuples
To: Anton Lindström
Added the message grammar and cleaned up the pair grammar.
It is only part of the build and not included in the
installation/distribution. Since it just loads multiple versions of the
same test sandbox it is only really useful for debugging during development.
Trink
On Mon, Feb 9, 2015 at 10:13 AM, Djamel F. wrote:
> recently I tried to load many sand
The current release as Kafka support.
Trink
On Tue, Feb 10, 2015 at 7:49 AM, Adriano Santos
wrote:
> Hi All,
>
> Does anybody know when a new release of heka including Kafka will be
> out?
>
> Best,
> Adriano
>
> ___
> Heka mailing list
> Heka@moz
ut?
>
> Best,
> Adriano
>
> On Tue, Feb 10, 2015 at 10:10 AM, Michael Trinkala
> wrote:
>
>> The current release as Kafka support.
>>
>> Trink
>>
>> On Tue, Feb 10, 2015 at 7:49 AM, Adriano Santos > > wrote:
>>
>>> Hi All,
>
loadstring is disallowed in the sandbox and that is unlikely to change.
There are other ways of decoding it but before we go there what is your use
case for creating a string suitable to load into the Lua interpreter? BTW:
raw strings will work properly in a Heka message as-is.
On Fri, Feb 13, 20
ing escaped strings you were referring to?
>
> Thanks.
>
>
>
> On 13 February 2015 at 20:04, Michael Trinkala
> wrote:
>
>> loadstring is disallowed in the sandbox and that is unlikely to change.
>>
>> There are other ways of decoding it but before we go there what
You could just add it to the decoder if desired. Using the Lua geoip module
it would look like this:
require "geoip.city"
local city_db = assert(geoip.city.open(read_config("geoip_city_db")))
.
.
.
fields.country_code = city_db:query_by_addr(fields.remote_addr,
"country_code")
On Thu, Mar 5, 20
Just remove the capture group and the timestamp variable will contain the
value you want.
local timestamp = l.P"[" * dt.build_strftime_grammar("%a %b %d %H:%M:%S.%s
%Y") / dt.time_to_ns * "]"
if timestamp then msg.Timestamp = timestamp end
Trink
On Sat, Mar 14, 2015 at 8:58 PM, Madhukar Thota
w
better able to assist you.
Trink
-- Forwarded message --
From: Madhukar Thota
Date: Sat, Mar 14, 2015 at 10:04 PM
Subject: Re: [heka] Timestamp Conversion
To: Michael Trinkala
Getting following error:
error: FATAL: process_message() invalid key to 'next'
Here is my
Currently there is no way to access it, you would have to duplicate the
entry in the config section of the plugin. Please file a feature request
if desired, thanks.
Trink
On Mon, Mar 23, 2015 at 4:16 AM, bruno binet wrote:
> Hi,
>
> I'd like to know the ticker_interval which is configured from
Timestamp is the number on nanoseconds since the Unix epoch. The
presentation is left to the consumer e.g., the RstEncoder will present it
as UTC.
Trink
On Fri, Apr 3, 2015 at 1:35 AM, Cristian Falcas
wrote:
> Hello,
>
> I started playing with heka and I'm seeing that the Timestamp field is
> s
The default limit is 8MiB, it can be adjusted with the memory_limit
configuration parameter.
https://hekad.readthedocs.org/en/latest/config/common_sandbox_parameter.html#config-common-sandbox-parameters
I would recommend setting it to at least 10x the size of your largest
document. The memory is
You should delete the build directory and rebuild everything from scratch.
Heka uses it own copy of Lua libluasb.so (this was recently renamed to
avoid any conflicts with an 'real' Lua installation). Please run 'ldd'
against lpeg.so and verify the module is linked to the correct library.
You shoul
Yeah, the apache decoder currently expects the data in the message
payload. Since your local Heka instance is sending the logs protobuf
encoded to the remote Heka (with the full log line in the message payload)
this can be achieved by using a multi decoder with a protobuf decoder first
or by modif
Use a fully qualified path for log_directory. I opened an issue
https://github.com/mozilla-services/heka/issues/1498
Trink
On Thu, Apr 23, 2015 at 11:12 AM, Amit Kumar wrote:
> Hello Team,
>
> I am new to Heka and am trying to learn what it has to offer for my data
> collection need. It appear
It is a bug, an issue has been filed
https://github.com/mozilla-services/heka/issues/1502.
Trink
On Tue, Apr 28, 2015 at 4:08 AM, briche arnaud
wrote:
> Hi all,
>
> I'm trying to setup two hekad instances, one instance forward syslog
> messages to the other using tcp input/output.
>
> The send
Yeah, we should address that:
https://github.com/mozilla-services/heka/issues/1514
On Tue, May 5, 2015 at 9:48 AM, Antonin Kral wrote:
> Hi Mathieu,
>
> glad I am not alone :)
>
> * Mathieu Chouquet-Stringer [2015-05-05 15:15] wrote:
> > Wait, I just discovered something, if I switch the offset
You may want to take a look at
https://github.com/mozilla-services/data-pipeline/blob/master/heka/plugins/s3splitfile/s3splitfile_output.go
Trink
On Wed, May 13, 2015 at 3:42 AM, Alex Jiao wrote:
> Hi, I'm writing an S3 output plugin for heka. As I am new to golang and
> heka, I'm not too sure
- It is usable and being actively developed with the intent to move it into
production later this year.
- We are currently running production data through it for testing but it is
not deployed in an official capacity. It has been very stable but until a
more robust set of tests have been build out
I wouldn't blame that on Lua. The (heavily used) standard printf %g
specifier generates output like that and Influx has chosen not to work with
it. I would push back on Influx, if a row is typed as float is should
still be able to accept an integer. If they are inferring type and you
send a 3 fi
os.date("%Y-%m-%dT%H:%M:%SZ", read_message("Timestamp") / 1e9)
On Thu, Jul 2, 2015 at 1:10 PM, Ali wrote:
> Thanks, Jarek!
>
> Those are only useful if I'm writing a Go plug-in, correct? Is there any
> way I can utilize such fancy time formatting libraries with a Lua
> encoder/decoder?
>
> My b
I can compare the performance on a
>> raspberry pi.
>> (here is the current state of our Heka-based data pipeline:
>> https://bitbucket.org/helioslite/heka-hl-sandboxes)
>> So it would be great if you can give me the first instructions on how to
>> build and setup Hindsig
t;
> On 7 July 2015 at 16:17, Michael Trinkala wrote:
>
>> Yeah, I have only been building on Ubuntu and haven't done any cross
>> platform clean-up. Thanks for the build output I will fix those errors
>> (unless you already have a patch).
>>
>> Tr
Feedback added to the gist.
Trink
On Tue, Jul 7, 2015 at 12:06 PM, Rob Miller wrote:
> On 07/07/2015 11:40 AM, Ali wrote:
>
>> Hi, all!
>>
>> I'm finally done with my Lua decoder and thought I would post a link to
>> it here, both for any constructive criticism anyone might have and for
>> help
Yeah, that VM was in a bad state I couldn't even ssh in. The reboot worked
though so it is back up.
Trink
On Fri, Jul 10, 2015 at 8:54 AM, Ali wrote:
> Hey, all.
>
> lpeg.trink.com seems to be down. Has anyone set up a publicly accessible
> instance of it?
>
> I'll see if I can set one up on
FYI: https://github.com/mozilla-services/heka/issues/826
On Mon, Aug 3, 2015 at 2:58 PM, Paul Bonser wrote:
> I've written a Go JSON decoder which allows you to configure which JSON
> fields go into which Heka message struct fields. The rest of the fields get
> put in the Heka message Fields map
gt; Sure, I will try your branch and report possible new compilation issues
>> in github.
>>
>> Cheers,
>> Bruno
>>
>> On 7 July 2015 at 18:26, Michael Trinkala wrote:
>>
>>> I changed the checkpoint id to an unsigned long long. Can you t
I change my output plugin to use the decode_message api function:
> 1439282566555139226 [error] output_plugins terminated:
> output/encode_metric.cfg msg: process_message()
> _hl/output/encode_metric.lua:15: attempt to call global 'decode_message' (a
> nil value)
>
>
> Thanks,
g I would just roll them after several minutes of what would be
average data flow on your system)
Trink
On Wed, Aug 12, 2015 at 1:55 AM, bruno binet wrote:
>
>
> On 12 August 2015 at 10:19, bruno binet wrote:
>
>> Thanks for all this valuable information.
>>
>
ovide a debian package for Hindsight to ease
> installation?
>
> Thanks for the great help!
> Bruno
>
>
>>
>> Trink
>>
>>
>>
>>
>>
>> On Wed, Aug 12, 2015 at 1:55 AM, bruno binet
>> wrote:
>>
>>>
>>>
>>&
ice without facing any performance issues anymore. :-)
>
> The last thing that I would like to fix before pushing Hindsight in
> production is to be able to compile the master branch on Raspbery PI ARM
> platform (for now I monkey patch the code for the compilation to succeed),
> see github i
Did you set the decoder = "apache" in the LogstreamerInput configuration?
Trink
On Wed, Sep 23, 2015 at 1:16 AM, Cristian Falcas
wrote:
> Hello,
>
> I can't manage to make heka parse any apache logs.
>
> Here is my configuration:
>
> ## apache config
> LogFormat "%h %l %u %t \"%r\" %>s %b \"%{R
Always, it is by design as described in the docs for decode_message:
https://hekad.readthedocs.org/en/latest/sandbox/index.html
- message (table) The array based version of the message structure with
the value member always being an array (even if there is only a single
item). This forma
Try: priority = ["Year", "Month", "Day", "^Seq"]
See:
https://hekad.readthedocs.org/en/latest/pluginconfig/logstreamer.html#single-sequential-rotating-logfile
Trink
On Mon, Oct 19, 2015 at 6:37 AM, Timur Batyrshin wrote:
> Hi all,
>
>
> I have the following list of logfiles:
>
> # heka-logstr
t)
> /var/log/tcollector/metrics.log
> /var/log/tcollector/metrics.log-10-18-2015-2.gz
> /var/log/tcollector/metrics.log-10-18-2015-1.gz
> /var/log/tcollector/metrics.log-10-19-2015-3.gz
> /var/log/tcollector/metrics.log-10-19-2015-2.gz
> /var/log/tcollector/metrics.log-10-19-
It is probably not the grammar failing, I am betting you return -1
somewhere else in the decoder. Can you share the code?
Trink
On Wed, Oct 21, 2015 at 11:23 AM, Justin Vogt
wrote:
> Hello Heka Community,
>
> I’m getting a weird error… I’ve written a custom decoder in Lua, and it
> works find
tname then
> msg.Hostname = flds.hostname
> flds.hostname = nil
> end
>
> msg.Payload = log
> msg.Fields = flds
>
> if not pcall(inject_message, msg) then return -1 end
> return 0
> end
>
> I tried a few more things, but sti
Coroutines can be enabled (3 lines above the link Rob sent, same type of
modification as the io tweak)
Trink
On Thu, Nov 19, 2015 at 1:50 AM, Kipras Mancevičius
wrote:
> Hi, thanks for all the info. I'll look into the options again. Most
> probably going to recompile Heka and enable the "io" mo
Please don't mistake the small size of the documentation as it being
incomplete. What is there is a full reference to the current functionality.
Its small size (when compared to the Heka documentation) is due to the
reduction of complexity in it's design (Hindsight is just a skeleton around
the He
Kafka - done (heavily used in our production setup and has been issue free
compared to the Go version)
https://mozilla-services.github.io/lua_sandbox_extensions/kafka/index.html
HttpInput - easy (just use the lua socket HTTP client)
HttpListenInput - pita (no HTTP server is currently available to t
We should probably be using the Hindsight list https://mail.mozilla.org/
listinfo/hindsight for this but I will provide a short answer here:
Hindsight is agnostic when it comes to push or pull in an I/O plugin. It
is a function of the plugin and there are currently examples of both.
However, as Er
It depends on the application but there can be benefits see (it has some
performance comparisons): http://www.lua.org/pil/9.4.html
Trink
On Wed, Sep 21, 2016 at 1:20 AM, Mathieu Parent
wrote:
> Hi,
>
> I'm currently trying to write a "syslog tcp" input. I've read heka_tcp
> input as inspiratio
Prometheus. If you wanted to go the other way the output plugin would be
an HTTP client that POSTs the data to an HTTP server.
Trink
On Wed, Sep 21, 2016 at 7:06 AM, Mathieu Parent
wrote:
> 2016-09-21 15:57 GMT+02:00 Michael Trinkala :
> > We should probably be using the Hindsight list
75 matches
Mail list logo
