On Wed, Jun 4, 2014 at 7:31 PM, Rob Miller <[email protected]> wrote:
> On Wed 04 Jun 2014 03:13:41 AM PDT, Victor Castell wrote:
>
>> Good to know, thank you.
>>
>> I'm using the "payload" format for ES output plugin, how this change
>> will affect me?
>>
>
> Good question. Every record that is sent to ElasticSearch needs to contain
> some BulkAPI JSON that precedes the document JSON. This tells ElasticSearch
> how the document is to be indexed, and typically looks something like this:
>
> {"index": {"_index": "YOUR_INDEX", "_type": "DOCUMENT_TYPE"}}
>
> It is the encoder's job to provide this. So to replicate the v0.5.2
> payload format, you'd create a very simple SandboxEncoder that generates
> this JSON and spits it out w/ the message payload immediately following.
> We've got a ticket open (https://github.com/mozilla-
> services/heka/issues/875) that I plan on working on today that will help
> w/ this. Once that's done, I'll send another message out w/ an example 0.6
> config replacing the use of payload format.
>
That's great, I'm open to help with this, we can follow the conversation in
the issue.
>
> -r
>
>
> On Wed, Jun 4, 2014 at 2:18 AM, Rob Miller <[email protected]
>> <mailto:[email protected]>> wrote:
>>
>> Hi,
>>
>> Just want to let everyone know that I've been working on
>> converting the ElasticSearchOutput from using its custom 'format'
>> configuration setting to instead using the new encoder plugin
>> infrastructure that will be in the upcoming 0.6 release.
>>
>> In v0.5.2, the two most (only?) used formats were "clean"
>> (generating a nicely formatted JSON representation of each
>> message) and "logstash_v0" (generating a JSON representation
>> matching the original "v0" schema used by Logstash). In 0.6, there
>> will be an ESJsonEncoder and an ESLogstashV0Encoder which generate
>> the same output, delta a few bug fixes.
>>
>> In the process, the config options for the ElasticSearchOutput
>> itself were cleaned up a bit.
>>
>> To give you an idea about the changes to expect, a v0.5.2 config
>> section like this:
>>
>> [ElasticSearchOutput]
>> message_matcher = "Type == 'nginx.access'"
>> index = "%{Type}-%{2006.01.02}"
>> type_name = "%{Type}"
>> server = "http://es-server:9200";
>> format = "clean"
>> esindexfromtimestamp = true
>> flush_interval = 5000
>> flush_count = 10
>>
>> Would in v0.6 look like this:
>>
>> [ESJsonEncoder]
>> index = "%{Type}-%{2006.01.02}"
>> es_index_from_timestamp = true
>> type_name = "%{Type}"
>>
>> [ElasticSearchOutput]
>> message_matcher = "Type == 'nginx.access'"
>> encoder = "ESJsonEncoder"
>> server = "http://es-server:9200";
>> flush_interval = 5000
>> flush_count = 10
>>
>> This change will be merged to dev momentarily. As always, feel
>> free to ask questions on this mailing list or in one of our #heka
>> IRC channels (on irc.freenode.net <http://irc.freenode.net> or
>> irc.mozilla.org <http://irc.mozilla.org>) if you have any issues.
>>
>> Thanks for Heka-ing!
>>
>> -r
>> _________________________________________________
>> Heka mailing list
>> [email protected] <mailto:[email protected]>
>> https://mail.mozilla.org/__listinfo/heka
>> <https://mail.mozilla.org/listinfo/heka>
>>
>>
>>
>>
>> --
>> V
>>
>
--
V
_______________________________________________
Heka mailing list
[email protected]
https://mail.mozilla.org/listinfo/heka
