That looks like there is an error on the ES side of things, check the logs
there. (the "no local error logger defined" line)
David Lang
On Tue, 7 Oct 2014, SjirBagmeijer wrote:
Thank you for all the help I got much further now, it seems it was indeed that
I was supposed to use $programname instead of the $Tag, now it looks like it
at-least connects to my elasticsearch according to the debug information:
7794.801098530:7f38231c4700: omelasticsearch: beginTransaction
7794.801103695:7f38231c4700: Action 0x7f382d70c650 transitioned to state: itx
7794.801108636:7f38231c4700: entering actionCalldoAction(), state: itx
7794.801171315:7f38231c4700: omelasticsearch: using REST URL:
'http://loghost.ulyaoth.net:9200/logstash-2014.10.07/events?'
7794.801772371:7f38231c4700: omelasticsearch: pData replyLen = '108'
7794.801782872:7f38231c4700: omelasticsearch: pData reply:
'{"_index":"logstash-2014.10.07","_type":"events","_id":"BHzZDcXTTjqim0mLsglobA","_version":1,"created":true}'
7794.801792573:7f38231c4700: omelasticsearch: no local error logger defined -
ignoring ES error information
7794.801797633:7f38231c4700: omelasticsearch: result doAction: 0 (bulkmode 0)
7794.801801664:7f38231c4700: Action 0x7f382d70c650 transitioned to state: rdy
7794.801805504:7f38231c4700: action 0x7f382d70c650 call returned 0
I cannot find it yet in Kibana somehow but I think I will be able to figure
that out :).
-----Original Message-----
From: "Radu Gheorghe"<[email protected]>
To: "rsyslog-users"<[email protected]>;
Cc:
Sent: 2014-10-07 (Tue) 22:15:31
Subject: Re: [rsyslog] json files directly to ES
Two more points from me that will hopefully help:
- if you're not sure where something breaks, try to isolate the problem by
reducing the config to the bare minimum and building up on it once it
works. For example, I wouldn't bother with rulesets if no logs can get to
ES in the first place. Just make sure you get your messages in and through
to ES. If a minimal config fails, usually running rsyslog -dn like David
advised should reveal the issue (for example, give you any exceptions ES
generates or libcurl errors)
- if you're sure logs are already JSON, I wouldn't bother parsing them. I
would just use templates to use the JSON as it is and eventually enrich it
with new properties. Take a look here (scroll down to the last section) for
an example:
http://wiki.rsyslog.com/index.php/Queues_on_v6_with_omelasticsearch
It's outdated (uses old config format for most snippets) but it should give
you some clues.
Best regards,
Radu
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/
On Tue, Oct 7, 2014 at 3:33 PM, David Lang <david>@lang.hm> wrote:
> you aren't showing us the entire config, so some of this is guesswork.
>
> try logging the data with the format RSYSLOG_DebugFormat to see what
> values are in each variable, the most common problem for things going to
> the wrong place is that the variables you are testing don't have the value
> you expect.
>
> I think you need to test $programname not $Tag (Tag is what you set in the
> infile input, but it gets put in the $programname variable)
>
> Beyond that, I would look at the eleasticsearch logs to see if it's
> complaining when you try to deliver the log.
>
> You can also start rsyslog in debug mode (-dn) to see all the gory details
> of what it's doing, you should see it attempting to deliver the log and any
> error it gets back.
>
> David Lang
>
>
> On Tue, 7 Oct 2014, SjirBagmeijer wrote:
>
> Hello,
>>
>> I have a small question I am trying to make rsyslog to sent log files
>> that are already formatted in json directly into my elasticsearch but I
>> have some trouble to get this to work is there someone that could
perhaps
>> see where I am going wrong with my config?
>>
>> Example of a log file content:
>> { "@timestamp": "2014-10-02T13:55:31+02:00", "message": "127.0.0.1 - -
>> [02/Oct/2014:13:55:31 +0200] \"GET /_status HTTP/1.1\" 401 38 \"-\"
>> \"curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.15.3
>> zlib/1.2.3 libidn/1.18 libssh2/1.4.2\"", "tags": ["nginx"], "clientip":
>> "127.0.0.1", "remote_user": "-", "contenttype": "text/html;
charset=utf-8",
>> "bytes": 38, "duration": "0.012", "status": "401", "request": "GET
/_status
>> HTTP/1.1", "method": "GET", "referrer": "-", "useragent": "curl/7.19.7
>> (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.15.3 zlib/1.2.3
libidn/1.18
>> libssh2/1.4.2" }
>>
>> Here below some examples that I tried to make by combining examples
from
>> you website:
>> http://www.rsyslog.com/tag/guides-for-rsyslog/
>> http://www.rsyslog.com/tag/elasticsearch/
>> http://www.rsyslog.com/doc/rsyslog_conf_filter.html
>>
>> I tried the following:
>> module(load="imfile" PollingInterval="10")
>> module(load="omelasticsearch")
>>
>> template(name="logstash-index"
>> type="list") {
>> constant(value="logstash-")
>> property(name="timereported" dateFormat="rfc3339" position.from="1"
>> position.to="4")
>> constant(value=".")
>> property(name="timereported" dateFormat="rfc3339" position.from="6"
>> position.to="7")
>> constant(value=".")
>> property(name="timereported" dateFormat="rfc3339" position.from="9"
>> position.to="10")
>> }
>>
>> template(name="jsonULY" type="list") {
>> property(name="$!all-json")
>> }
>>
>> ruleset(name="logstash"){
>> action(type="omelasticsearch"
>> server="loghost.ulyaoth.net"
>> serverport="9200"
>> searchIndex="logstash-index"
>> dynSearchIndex="on"
>> template="jsonULY")
>> stop
>> }
>>
>>
>> input(type="imfile"
>> File="/var/log/nginx/access.json"
>> Tag="accessnginx"
>> StateFile="/var/spool/rsyslog/accessnginx"
>> ruleset="logstash")
>>
>> This seems to do nothing at all somehow, then I also tried the
following:
>> template(name="jsonULY" type="list") {
>> property(name="$!all-json")
>> }
>>
>> input(type="imfile"
>> File="/var/log/nginx/access.json"
>> Tag="nginxaccess"
>> StateFile="/var/spool/rsyslog/nginxaccess")
>>
>> if $Tag == 'nginxaccess' then {
>>
>> action(type="omelasticsearch"
>> server="logstash.ulyaoth.net"
>> serverport="9200"
>> searchIndex="logstash-index"
>> dynSearchIndex="on"
>> template="jsonULY")
>> stop
>> }
>>
>> And multiple other ways, it seems the input works but most of the time
it
>> is going directly to my /var/log/messages instead of going to my ES. I
also
>> double checked that my ports are open.
>>
>> Is there someone that or can see what I do wrong and give me a hint, or
>> perhaps a link to some example where people sent already existing json
>> files directly to ES with rsyslog?
>>
>> Thank you so much in advanced.
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>>
>> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
