Elasticsearch does not crash. You might be confusing this with another
email thread I just started where *Rsyslog* process crashes because of trap
divide / segfault.
I do have the full payload that Elasticsearch is trying to process
and "\u00E0\u00.4�\" is the character sequence around character 28863.
I think the root cause of the problem is that there is an invalid UTF-8
sequence "\u00.4" in the value if the "message" field. In fact, I just
confirmed that {"message":"\u00.4"} is not a valid JSON on
http://jsonlint.com/.
So, the problem seems to lie in whatever plugin is generating this JSON on
the Rsyslog side. Would you agree?
Thanks,
Alec
On Tue, Jun 28, 2016 at 2:30 PM, David Lang <[email protected]> wrote:
> On Tue, 28 Jun 2016, Alec Swan wrote:
>
> Thanks, David. I actually used that set/unset trick in the past with
>> event.tags :)
>>
>> I am including the full exception stack trace which makes me believe that
>> the exception is thrown while parsing the content of the "message" field
>> which is extracted by one of my liblognorm rules. The value of this field
>> is rather long and it does have a somewhat strange character sequence
>> around character 28863 "\u00E0\u00.4???\". Do you think this could be the
>> root cause of this exception?
>>
>
> it's possible, but the error message you are providing sure seems like
> it's complaining about '.' (which is ascii 46)
>
> do you have any way of finding out what the full string is that it's
> trying to process?
>
> have you asked elastic.io? (since the problem seems to be in their end),
> even if it's invalid json, they shouldn't crash, just reject the message.
>
> David Lang
>
>
> MapperParsingException[failed to parse [message]]; nested:
>> JsonParseException[Unexpected character ('.' (code 46)): expected a
>> hex-digit for character escape sequence
>> at [Source:
>> org.elasticsearch.common.io.stream.InputStreamStreamInput@17437531; line:
>> 1, column: 28863]];
>> at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:329)
>> at
>>
>> org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:309)
>> at
>>
>> org.elasticsearch.index.mapper.DocumentParser.parseValue(DocumentParser.java:436)
>> at
>>
>> org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:262)
>> at
>>
>> org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:122)
>> at
>>
>> org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:309)
>> at
>>
>> org.elasticsearch.index.shard.IndexShard.prepareCreate(IndexShard.java:529)
>> at
>>
>> org.elasticsearch.index.shard.IndexShard.prepareCreateOnPrimary(IndexShard.java:506)
>> at
>>
>> org.elasticsearch.action.index.TransportIndexAction.prepareIndexOperationOnPrimary(TransportIndexAction.java:215)
>> at
>>
>> org.elasticsearch.action.index.TransportIndexAction.executeIndexRequestOnPrimary(TransportIndexAction.java:224)
>> at
>>
>> org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:326)
>> at
>>
>> org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:119)
>> at
>>
>> org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:68)
>> at
>>
>> org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase.doRun(TransportReplicationAction.java:639)
>> at
>>
>> org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
>> at
>>
>> org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:279)
>> at
>>
>> org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:271)
>> at
>>
>> org.elasticsearch.shield.transport.ShieldServerTransportService$ProfileSecuredRequestHandler.messageReceived(ShieldServerTransportService.java:180)
>> at
>>
>> org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75)
>> at
>>
>> org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:376)
>> at
>>
>> org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
>> at
>>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>> Caused by: com.fasterxml.jackson.core.JsonParseException: Unexpected
>> character ('.' (code 46)): expected a hex-digit for character escape
>> sequence
>> at [Source:
>> org.elasticsearch.common.io.stream.InputStreamStreamInput@17437531; line:
>> 1, column: 28863]
>> at
>>
>> com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1581)
>> at
>>
>> com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:533)
>> at
>>
>> com.fasterxml.jackson.core.base.ParserMinimalBase._reportUnexpectedChar(ParserMinimalBase.java:462)
>> at
>>
>> com.fasterxml.jackson.core.json.UTF8StreamJsonParser._decodeEscaped(UTF8StreamJsonParser.java:3188)
>> at
>>
>> com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishString2(UTF8StreamJsonParser.java:2459)
>> at
>>
>> com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishAndReturnString(UTF8StreamJsonParser.java:2414)
>> at
>>
>> com.fasterxml.jackson.core.json.UTF8StreamJsonParser.getText(UTF8StreamJsonParser.java:285)
>> at
>>
>> org.elasticsearch.common.xcontent.json.JsonXContentParser.text(JsonXContentParser.java:84)
>> at
>>
>> org.elasticsearch.common.xcontent.support.AbstractXContentParser.textOrNull(AbstractXContentParser.java:194)
>> at
>>
>> org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateFieldForString(StringFieldMapper.java:368)
>> at
>>
>> org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateField(StringFieldMapper.java:311)
>> at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:321)
>> ... 23 more
>>
>>
>> Thanks,
>>
>> Alec
>>
>> On Tue, Jun 28, 2016 at 11:36 AM, David Lang <[email protected]> wrote:
>>
>> This is actually valid JSON, but ElasticSearch (as of v2) has decided that
>>> since Kibana uses '.' as a delimeter between fields, that ElasticSearch
>>> would disallow '.' in field names.
>>>
>>> There is not currently a plugin to go through and sanitize field names.
>>> It's been requested for a while, but hasn't bubbled up to the top of
>>> anyone's to-do list yet. I got a quote to implement this (500 euro), but
>>> my
>>> company got tangled in internal politics on the issue and is not going to
>>> be able to do so anytime soon.
>>>
>>> In the meantime you can use the set and unset commands to manually rename
>>> the fields that your logs have that have . in them
>>>
>>> for example, for the event.tags created by liblognorm you can do:
>>>
>>> set event_tags = event.tags;
>>> unset event.tags;
>>>
>>> This isn't a good work-around, but it does work. If anyone is able to
>>> drum
>>> up sponsorship for the feature, it can be available pretty quickly.
>>>
>>> David Lang
>>>
>>> On Tue, 28 Jun 2016, Alec Swan wrote:
>>>
>>> Hello,
>>>
>>>>
>>>> I am using rsyslog 8.19.0 with mmnorlamize and liblognorn to parse our
>>>> log
>>>> messages and convert them to JSON. After that I use omelaticsearch to
>>>> send
>>>> JSON to Elasticsearch 2.2.3.
>>>>
>>>> I noticed the following recurring error in the Elasticsearch logs, which
>>>> makes me believe that one of the plugins listed above is producing
>>>> invalid
>>>> JSON. Which plugin would that be and is there a fix for this?
>>>>
>>>> *Unexpected character ('.' (code 46)): expected a hex-digit for
>>>> character escape sequence*
>>>>
>>>> Thanks,
>>>>
>>>> Alec
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>>> DON'T LIKE THAT.
>>>>
>>>> _______________________________________________
>>>>
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>> DON'T LIKE THAT.
>>>
>>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
