Thanks, David. I actually used that set/unset trick in the past with
event.tags :)
I am including the full exception stack trace which makes me believe that
the exception is thrown while parsing the content of the "message" field
which is extracted by one of my liblognorm rules. The value of this field
is rather long and it does have a somewhat strange character sequence
around character 28863 "\u00E0\u00.4�\". Do you think this could be the
root cause of this exception?
MapperParsingException[failed to parse [message]]; nested:
JsonParseException[Unexpected character ('.' (code 46)): expected a
hex-digit for character escape sequence
at [Source:
org.elasticsearch.common.io.stream.InputStreamStreamInput@17437531; line:
1, column: 28863]];
at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:329)
at
org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:309)
at
org.elasticsearch.index.mapper.DocumentParser.parseValue(DocumentParser.java:436)
at
org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:262)
at
org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:122)
at
org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:309)
at
org.elasticsearch.index.shard.IndexShard.prepareCreate(IndexShard.java:529)
at
org.elasticsearch.index.shard.IndexShard.prepareCreateOnPrimary(IndexShard.java:506)
at
org.elasticsearch.action.index.TransportIndexAction.prepareIndexOperationOnPrimary(TransportIndexAction.java:215)
at
org.elasticsearch.action.index.TransportIndexAction.executeIndexRequestOnPrimary(TransportIndexAction.java:224)
at
org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:326)
at
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:119)
at
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:68)
at
org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase.doRun(TransportReplicationAction.java:639)
at
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at
org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:279)
at
org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:271)
at
org.elasticsearch.shield.transport.ShieldServerTransportService$ProfileSecuredRequestHandler.messageReceived(ShieldServerTransportService.java:180)
at
org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75)
at
org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:376)
at
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.fasterxml.jackson.core.JsonParseException: Unexpected
character ('.' (code 46)): expected a hex-digit for character escape
sequence
at [Source:
org.elasticsearch.common.io.stream.InputStreamStreamInput@17437531; line:
1, column: 28863]
at
com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1581)
at
com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:533)
at
com.fasterxml.jackson.core.base.ParserMinimalBase._reportUnexpectedChar(ParserMinimalBase.java:462)
at
com.fasterxml.jackson.core.json.UTF8StreamJsonParser._decodeEscaped(UTF8StreamJsonParser.java:3188)
at
com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishString2(UTF8StreamJsonParser.java:2459)
at
com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishAndReturnString(UTF8StreamJsonParser.java:2414)
at
com.fasterxml.jackson.core.json.UTF8StreamJsonParser.getText(UTF8StreamJsonParser.java:285)
at
org.elasticsearch.common.xcontent.json.JsonXContentParser.text(JsonXContentParser.java:84)
at
org.elasticsearch.common.xcontent.support.AbstractXContentParser.textOrNull(AbstractXContentParser.java:194)
at
org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateFieldForString(StringFieldMapper.java:368)
at
org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateField(StringFieldMapper.java:311)
at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:321)
... 23 more
Thanks,
Alec
On Tue, Jun 28, 2016 at 11:36 AM, David Lang <[email protected]> wrote:
> This is actually valid JSON, but ElasticSearch (as of v2) has decided that
> since Kibana uses '.' as a delimeter between fields, that ElasticSearch
> would disallow '.' in field names.
>
> There is not currently a plugin to go through and sanitize field names.
> It's been requested for a while, but hasn't bubbled up to the top of
> anyone's to-do list yet. I got a quote to implement this (500 euro), but my
> company got tangled in internal politics on the issue and is not going to
> be able to do so anytime soon.
>
> In the meantime you can use the set and unset commands to manually rename
> the fields that your logs have that have . in them
>
> for example, for the event.tags created by liblognorm you can do:
>
> set event_tags = event.tags;
> unset event.tags;
>
> This isn't a good work-around, but it does work. If anyone is able to drum
> up sponsorship for the feature, it can be available pretty quickly.
>
> David Lang
>
> On Tue, 28 Jun 2016, Alec Swan wrote:
>
> Hello,
>>
>> I am using rsyslog 8.19.0 with mmnorlamize and liblognorn to parse our log
>> messages and convert them to JSON. After that I use omelaticsearch to send
>> JSON to Elasticsearch 2.2.3.
>>
>> I noticed the following recurring error in the Elasticsearch logs, which
>> makes me believe that one of the plugins listed above is producing invalid
>> JSON. Which plugin would that be and is there a fix for this?
>>
>> *Unexpected character ('.' (code 46)): expected a hex-digit for
>> character escape sequence*
>>
>> Thanks,
>>
>> Alec
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>>
>> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
