Hello,
I have a small question I am trying to make rsyslog to sent log files that are
already formatted in json directly into my elasticsearch but I have some
trouble to get this to work is there someone that could perhaps see where I am
going wrong with my config?
Example of a log file content:
{ "@timestamp": "2014-10-02T13:55:31+02:00", "message": "127.0.0.1 - -
[02/Oct/2014:13:55:31 +0200] \"GET /_status HTTP/1.1\" 401 38 \"-\"
\"curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.15.3 zlib/1.2.3
libidn/1.18 libssh2/1.4.2\"", "tags": ["nginx"], "clientip": "127.0.0.1",
"remote_user": "-", "contenttype": "text/html; charset=utf-8", "bytes": 38,
"duration": "0.012", "status": "401", "request": "GET /_status HTTP/1.1",
"method": "GET", "referrer": "-", "useragent": "curl/7.19.7
(x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.15.3 zlib/1.2.3 libidn/1.18
libssh2/1.4.2" }
Here below some examples that I tried to make by combining examples from you
website:
http://www.rsyslog.com/tag/guides-for-rsyslog/
http://www.rsyslog.com/tag/elasticsearch/
http://www.rsyslog.com/doc/rsyslog_conf_filter.html
I tried the following:
module(load="imfile" PollingInterval="10")
module(load="omelasticsearch")
template(name="logstash-index"
type="list") {
constant(value="logstash-")
property(name="timereported" dateFormat="rfc3339" position.from="1"
position.to="4")
constant(value=".")
property(name="timereported" dateFormat="rfc3339" position.from="6"
position.to="7")
constant(value=".")
property(name="timereported" dateFormat="rfc3339" position.from="9"
position.to="10")
}
template(name="jsonULY" type="list") {
property(name="$!all-json")
}
ruleset(name="logstash"){
action(type="omelasticsearch"
server="loghost.ulyaoth.net"
serverport="9200"
searchIndex="logstash-index"
dynSearchIndex="on"
template="jsonULY")
stop
}
input(type="imfile"
File="/var/log/nginx/access.json"
Tag="accessnginx"
StateFile="/var/spool/rsyslog/accessnginx"
ruleset="logstash")
This seems to do nothing at all somehow, then I also tried the following:
template(name="jsonULY" type="list") {
property(name="$!all-json")
}
input(type="imfile"
File="/var/log/nginx/access.json"
Tag="nginxaccess"
StateFile="/var/spool/rsyslog/nginxaccess")
if $Tag == 'nginxaccess' then {
action(type="omelasticsearch"
server="logstash.ulyaoth.net"
serverport="9200"
searchIndex="logstash-index"
dynSearchIndex="on"
template="jsonULY")
stop
}
And multiple other ways, it seems the input works but most of the time it is
going directly to my /var/log/messages instead of going to my ES. I also double
checked that my ports are open.
Is there someone that or can see what I do wrong and give me a hint, or perhaps
a link to some example where people sent already existing json files directly
to ES with rsyslog?
Thank you so much in advanced.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
