Have been able to reproduce the problem locally using the existing test
cases inside ES Sink. The problem does exist.
Did some initial investigation, the framework is able to detect the JSON
content and tries to add it as complex field.
timestamp is added only if present in header.
In the class org.apache.flume.sink.elasticsearch.ContentBuilderUtil
public static void addComplexField(XContentBuilder builder, String
fieldName,
XContentType contentType, byte[] data) throws IOException {
XContentParser parser = null;
try {
XContentBuilder tmp = jsonBuilder();
parser = XContentFactory.xContent(contentType).createParser(data);
parser.nextToken();
tmp.copyCurrentStructure(parser);
builder.field(fieldName, tmp); <<<< This is where the we might have
an issue (real action is happening inside this method
call)
Can someone familiar with this part look further into this? I shall debug
further as soon as I have free cycles.
thanks
ashish
On Fri, Apr 11, 2014 at 5:24 PM, Deepak Subhramanian <
[email protected]> wrote:
> Thanks Simon. I am also struggling with no luck. I tried using the latest
> flume elastic search sink jar build from 1.5SNAPSHOT ,but still no luck. I
> will try to see if it is an issue with elastic search api . When I loaded
> json using hive it loaded JSON properly. But we have to pass a property
> es.input.json in hive. Is there a way to pass the same in Flume.
>
> CREATE EXTERNAL TABLE json (data STRING
> <http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/hive.html#CO25-1>)
>
> STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
> TBLPROPERTIES('es.resource' = '...',
>
> 'es.input.json` = 'yes'
> <http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/hive.html#CO25-2>);
>
>
--
thanks
ashish
Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal
