Hello,
you can use format="json*" in order to avoid unnecessary escaping:
http://www.rsyslog.com/doc/property_replacer.html[1]
##CEE TEMPLATE
template(name="cee" type="list") {
constant(value="<") property(name="pri") constant(value=">")
property(name="timereported" dateFormat="rfc3339")
constant(value=" ") property(name="$myhostname")
constant(value=" ") property(name="programname")
constant(value=" ")
constant(value="@cee: {")
#SYSLOG
property(name="$myhostname" format="jsonf" outname="host")
constant(value=", ")
property(name="syslogtag" format="jsonf" outname="tag") constant(value=",
")
property(name="programname" format="jsonf" outname="prog")
constant(value=", ")
property(name="syslogfacility-text" format="jsonf" outname="facility")
constant(value=", ")
property(name="syslogpriority-text" format="jsonf" outname="priority")
constant(value=", ")
property(name="timereported" dateFormat="rfc3339" format="jsonf"
outname="syslog_timestamp") constant(value=", ")
##ES TIMESTAMP
constant(value="\"@timestamp\":\"")
property(name="timegenerated" dateFormat="unixtimestamp")
property(name="timegenerated" dateFormat="subseconds" position.to="3")
constant(value="\", ")
#REST
property(name="$!all-json" position.from="2")
}
/---/
*/Best regards,/*
/Eugene Istomin/
/EDS Systems/
/[email protected]/
/Work: +372-6409-600/
> Hello all,
>
> I have some exceptions from elastic search regarding messages sent by
> rsyslog.
>
> Template in rsyslog used for elasticsearch:
>
> template(name="es_template"
> type="list"
> option.json="on")
> {
> constant(value="{")
> constant(value="\"@timestamp\":\"")
> property(name="timereported" dateFormat="rfc3339")
> constant(value="\",\"timereported\":\"")
> property(name="timereported" dateFormat="rfc3339")
> constant(value="\",\"timegenerated\":\"")
> property(name="timegenerated" dateFormat="rfc3339")
> constant(value="\",\"message\":\"") property(name="msg")
> constant(value="\",\"host\":\"")
> property(name="hostname")
> constant(value="\",\"severity\":\"")
> property(name="syslogseverity-text")
> constant(value="\",\"priority\":\"")
> property(name="syslogpriority-text")
> constant(value="\",\"facility\":\"")
> property(name="syslogfacility-text")
> constant(value="\",\"tag\":\"")
> property(name="syslogtag")
> constant(value="\",\"program_name\":\"")
> property(name="programname")
> constant(value="\"}")
> }
>
>
> Json sent by rsyslog:
>
> {
> "@timestamp": "2014-07-23T15:08:37.262843+03:00",
> "timereported": "2014-07-23T15:08:37.262843+03:00",
> "timegenerated": "2014-07-23T15:08:37.262843+03:00",
> "message": " File
> \"/opt/optymyze/collectd/scripts/python/web_plugin.py\", line 63, in
> read#012 metrics.type_instance = re.sub(\"\ \",\"-\",transactions[j]) +
> \"_response_time\"",
> "host": "is-iasi-vm.synygy.net",
> "severity": "err",
> "priority": "err",
> "facility": "daemon",
> "tag": "collectd[29370]:",
> "program_name": "collectd"
> }
>
> Elastic search complains about space character being escaped. This happens
> here:
> re.sub(\"\ \",\"-\"
>
>
>
> Exception log:
>
> [2014-07-23 15:08:37,273][DEBUG][action.bulk ]
> [v-so-repo-02-es-01] [default-index][1] failed to execute bulk item (index)
> index {[default-index][collectd][knQxDu2MR3eYgSiZc6TmLQ],
> source[{"@timestamp":"2014-07-23T15:08:37.262843+03:00","timereported":"2014
> -07-23T15:08:37.262843+03:00","timegenerated":"2014-07-23T15:08:37.262843+03
> :00","message":" File
> \"/opt/optymyze/collectd/scripts/python/web_plugin.py\", line 63, in
> read#012 metrics.type_instance = re.sub(\"\ \",\"-\",transactions[j]) +
> \"_response_time\"","host":"is-iasi-vm.synygy.net
> ","severity":"err","priority":"err","facility":"daemon","tag":"collectd[2937
> 0]:","program_name":"collectd"}]}
> org.elasticsearch.index.mapper.MapperParsingException: failed to parse
> [message]
> at
> org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldM
> apper.java:417) at
> org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapp
> er.java:637) at
> org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:4
> 90) at
> org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:515)
> at
> org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:462)
> at
> org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(Inter
> nalIndexShard.java:373) at
> org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(T
> ransportShardBulkAction.java:425) at
> org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrima
> ry(TransportShardBulkAction.java:158) at
> org.elasticsearch.action.support.replication.TransportShardReplicationOperat
> ionAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicati
> onOperationAction.java:534) at
> org.elasticsearch.action.support.replication.TransportShardReplicationOperat
> ionAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperation
> Action.java:433) at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
> 45) at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
> 15) at java.lang.Thread.run(Thread.java:744)
> Caused by: org.elasticsearch.common.jackson.core.JsonParseException:
> Unrecognized character escape ' ' (code 32)
> at [Source: [B@5f263984; line: 1, column: 286]
> at
> org.elasticsearch.common.jackson.core.JsonParser._constructError(JsonParser.
> java:1524) at
> org.elasticsearch.common.jackson.core.base.ParserMinimalBase._reportError(Pa
> rserMinimalBase.java:557) at
> org.elasticsearch.common.jackson.core.base.ParserMinimalBase._handleUnrecogn
> izedCharacterEscape(ParserMinimalBase.java:532) at
> org.elasticsearch.common.jackson.core.json.UTF8StreamJsonParser._decodeEscap
> ed(UTF8StreamJsonParser.java:2817) at
> org.elasticsearch.common.jackson.core.json.UTF8StreamJsonParser._finishStrin
> g2(UTF8StreamJsonParser.java:2193) at
> org.elasticsearch.common.jackson.core.json.UTF8StreamJsonParser._finishStrin
> g(UTF8StreamJsonParser.java:2149) at
> org.elasticsearch.common.jackson.core.json.UTF8StreamJsonParser.getText(UTF8
> StreamJsonParser.java:281) at
> org.elasticsearch.common.xcontent.json.JsonXContentParser.text(JsonXContentP
> arser.java:85) at
> org.elasticsearch.common.xcontent.support.AbstractXContentParser.textOrNull(
> AbstractXContentParser.java:194) at
> org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateFieldForStr
> ing(StringFieldMapper.java:338) at
> org.elasticsearch.index.mapper.core.StringFieldMapper.parseCreateField(Strin
> gFieldMapper.java:278) at
> org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldM
> apper.java:407) ... 12 more
>
> Should I open a bug for this, or is something wrong on my side?
>
> Best regards,
> Cristian Falcas
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
--------
[1] http://www.rsyslog.com/doc/property_replacer.html
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
