Additions to my earlier question:
1. It looks like I can customize which index to write into in the
prepareIndexREquest(...) method - when implementing my custom event serializer.
I could probably do something like this (very pseudo-code-ish):-- get
customerId from the Event [actually, where is the org.apache.flume.Event class?
I could not find it in the apache-flume-1.5.2-src.tar ...]-- create String
myIndexName = 'indexPrefix-from-config' + "_" + customerId-- in
[repareIndexRequest(....) do: indexRequestBuilder.setIndex(myIndexName).
....would this work?
2. However, I am not sure how and when this index is created.... ? I understand
that by default a new index with the configured indexPrefix will be created
every day - is it possible to customize this behavior?
3. And one more question: I see that it is the IndexREquestBuilder class that
is used for indexing, and not BulkRequestBuilder. Does it mean that Flume
indexes one event at a time only? without any batching?
thanks!Marina
From: Marina <[email protected]>
To: "[email protected]" <[email protected]>
Sent: Wednesday, April 15, 2015 4:06 PM
Subject: Flume ElasticSearchSink - how to customize index name
Hi,I'm researching am option of populating ElasticSearch cluster from Flume for
Apache-like logs. One of the requirements is to add log events to a specific
index - the index name has to be determined based on a 'customerId'-like
property available in the log message, that can be parsed out.I see that I can
potentially customize event parsing logic by writing a custom SErializer by
implementing the ElasticSearchIndexRequestBuilderFactory interface.Any
suggestion on how I could also customize index selection?
thanks!Marina
