[
https://issues.apache.org/jira/browse/FLUME-2126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14319535#comment-14319535
]
Xuri Nagarin commented on FLUME-2126:
-------------------------------------
I am using morphlines from Kite SDK to handle Json. My point is, sure you can
make a minor code change in the sink to handle Json and that will handle our
use case. But what if the body has some other structure format than Json?
Wouldn't you then have to add logic to first detect the structure (Json, XML
etc) and then apply corresponding processing to match what ES expects? That
might make the sink un-necessarily complex?
morphlines : [
{
id : morphline1
importCommands : ["org.kitesdk.**","org.apache.solr.**"]
commands : [
{
readJson {}
}
{
extractJsonPaths {
flatten : true
paths : {
appname : /appname
serviceType : /serviceType
authStatus : /authStatus
uuid : /uuid
service : /service
rawMsg : /rawMsg
clientSrcPort : /clientSrcPort
user : /user
hostname : /hostname
ts : /ts
sshProtocol : /sshProtocol
environment : /environment
logType : /logType
sshAuthMethod : /sshAuthMethod
rawTimestamp : /rawTimestamp
normalizedTimeStamp : /normalizedTimeStamp
regexMatch : /regexMatch
clientIp : /clientIp
syslogHostIp : /syslogHostIp
logAgent : /logAgent
}
}
}
{ generateUUID {
field : uuid
preserveExisting: true
}
}
{ setValues {_attachment_body : "@{rawMsg}" }}
{ toByteArray { field : _attachment_body} }
]
}
]
> Problem in elasticsearch sink when the event body is a complex field
> --------------------------------------------------------------------
>
> Key: FLUME-2126
> URL: https://issues.apache.org/jira/browse/FLUME-2126
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Environment: 1.3.1 and 1.4
> Reporter: Massimo Paladin
> Assignee: Ashish Paliwal
> Attachments: FLUME-2126-0.patch
>
>
> I have found a bug in the elasticsearch sink, the problem is in the
> {{ContentBuilderUtil.addComplexField}} method, when it does
> {{builder.field(fieldName, tmp);}} the {{tmp}} object is taken as {{Object}}
> with the result of being serialized with the {{toString}} method in the
> {{XContentBuilder}}. In the end you get the object reference as content.
> The following change workaround the problem for me, the bad point is that it
> has to parse the content twice, I guess there is a better way to solve the
> problem but I am not an elasticsearch api expert.
> {code}
> ---
> a/flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ContentBuilderUtil.java
> +++
> b/flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ContentBuilderUtil.java
> @@ -61,7 +61,12 @@ public class ContentBuilderUtil {
> parser = XContentFactory.xContent(contentType).createParser(data);
> parser.nextToken();
> tmp.copyCurrentStructure(parser);
> - builder.field(fieldName, tmp);
> +
> + // if it is a valid structure then we include it
> + parser = XContentFactory.xContent(contentType).createParser(data);
> + parser.nextToken();
> + builder.field(fieldName);
> + builder.copyCurrentStructure(parser);
> } catch (JsonParseException ex) {
> // If we get an exception here the most likely cause is nested JSON
> that
> // can't be figured out in the body. At this point just push it through
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
