little more on the issue builder.field(fieldName, tmp); calls the XContentBuilder API where class type is determined and appropriate method is called. Since tmp, which is instance of XContentBuilder, doesn't match any of the defined if conditions it goes to final else where the tmp.toString() is called, and field(String, String) method is called so we get object address in index.
Replacing builder.field(fieldName, tmp); with builder.field(fieldName, tmp.string()); shall make things work, but I am not sure if this would be the best way to use the API. Got the answer from ES user list :) http://elasticsearch-users.115913.n3.nabble.com/Issue-with-posting-json-data-to-elastic-search-via-Flume-td4054017.html Can ES experts comment on the best way forward? On Sun, Apr 13, 2014 at 8:10 PM, Ashish <[email protected]> wrote: > Have been able to reproduce the problem locally using the existing test > cases inside ES Sink. The problem does exist. > > Did some initial investigation, the framework is able to detect the JSON > content and tries to add it as complex field. > timestamp is added only if present in header. > > In the class org.apache.flume.sink.elasticsearch.ContentBuilderUtil > > public static void addComplexField(XContentBuilder builder, String > fieldName, > XContentType contentType, byte[] data) throws IOException { > XContentParser parser = null; > try { > XContentBuilder tmp = jsonBuilder(); > parser = XContentFactory.xContent(contentType).createParser(data); > parser.nextToken(); > tmp.copyCurrentStructure(parser); > builder.field(fieldName, tmp); <<<< This is where the we might have > an issue (real action is happening inside this method > call) > > Can someone familiar with this part look further into this? I shall debug > further as soon as I have free cycles. > > thanks > ashish > > > > On Fri, Apr 11, 2014 at 5:24 PM, Deepak Subhramanian < > [email protected]> wrote: > >> Thanks Simon. I am also struggling with no luck. I tried using the >> latest flume elastic search sink jar build from 1.5SNAPSHOT ,but still no >> luck. I will try to see if it is an issue with elastic search api . When I >> loaded json using hive it loaded JSON properly. But we have to pass a >> property es.input.json in hive. Is there a way to pass the same in Flume. >> >> CREATE EXTERNAL TABLE json (data STRING >> <http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/hive.html#CO25-1>) >> >> >> STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler' >> TBLPROPERTIES('es.resource' = '...', >> >> >> 'es.input.json` = 'yes' >> <http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/hive.html#CO25-2>); >> >> > > > -- > thanks > ashish > > Blog: http://www.ashishpaliwal.com/blog > My Photo Galleries: http://www.pbase.com/ashishpaliwal > -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal
