Hello and thank you once more for the responses, so I have been testing but it
seems whatever I try it indeed does not somehow save the log in ElasticSearch
even it does say it did.
I looked at ElasticSearch but there is no errors in the logs or any strange
behavior apart from this.
The output from the command below is as following:
[loghost ~]# curl localhost:9200/logstash-2014.10.07/_search?pretty
{
"took" : 78,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 8232067,
"max_score" : 1.0,
"hits" : [ {
"_index" : "logstash-2014.10.07",
"_type" : "events",
"_id" : "NWKSiq5NTvysGkVZ9OR8XA",
"_score" : 1.0, "_source" : {}
}, {
"_index" : "logstash-2014.10.07",
"_type" : "events",
"_id" : "-5wf5CK_R5iCe1RF0zzsPg",
"_score" : 1.0, "_source" : {}
}, {
"_index" : "logstash-2014.10.07",
"_type" : "events",
"_id" : "LB8gFrTaRGyI5YreIPBS9w",
"_score" : 1.0, "_source" : {}
}, {
"_index" : "logstash-2014.10.07",
"_type" : "events",
"_id" : "aCck3E1GTqeanVAeHGGDsg",
"_score" : 1.0, "_source" : {}
}, {
"_index" : "logstash-2014.10.07",
"_type" : "events",
"_id" : "uCAMM4TGRD205AEMZMWtTQ",
"_score" : 1.0, "_source" : {}
}, {
"_index" : "logstash-2014.10.07",
"_type" : "events",
"_id" : "5JzfuFyXRiCxhhj73A249w",
"_score" : 1.0, "_source" : {}
}, {
"_index" : "logstash-2014.10.07",
"_type" : "events",
"_id" : "zNHmX6udT5GB7z5H2qD5bw",
"_score" : 1.0, "_source" : {}
}, {
"_index" : "logstash-2014.10.07",
"_type" : "events",
"_id" : "x_-nDe19SCCIuOKsd92CWw",
"_score" : 1.0, "_source" : {}
}, {
"_index" : "logstash-2014.10.07",
"_type" : "events",
"_id" : "s9wg8HhnRA6XSTf2kRMc1A",
"_score" : 1.0, "_source" : {}
}, {
"_index" : "logstash-2014.10.07",
"_type" : "events",
"_id" : "TGDNq7udT3uU5r-W6uI9TQ",
"_score" : 1.0, "_source" : {}
} ]
}
}
It looks to me the logs somehow is not saved, I tried use some other rsyslog
configuration by shipping the logs to a tcp port and then let logstash catch it
this works without issues by doing this:
if $programname == 'default-nginx-accesslog' then @loghost:5544
if $programname == 'default-nginx-accesslog' then ~
But this way it seems I have to grok all the logs correctly, any idea what else
I could look? Is there perhaps a version requirement of ElasticSearch in order
to sent it directly to it?
I currently use: (rpms on rhel7)
elasticsearch-1.1.1-1
logstash-1.4.1-1
Thanks again,
Sjir Bagmeijer
-----Original Message-----
From: "Radu Gheorghe"<[email protected]>
To: "rsyslog-users"<[email protected]>;
Cc:
Sent: 2014-10-08 (Wed) 01:31:07
Subject: Re: [rsyslog] json files directly to ES
ES is fine, this means the document was successfully indexed:
{"_index":"logstash-2014.10.07","_type":"events","_id":"BHzZDcXTTjqim0mLsglobA","_version":1,"created":true}
The index name looks fine to me, I'm not sure why you can't see it in
Kibana. Maybe you can extend the timeframe? (I had issues when my system's
clock was behind a bit).
If you can't figure it out, let's try the old-school way and post here the
output of this command on one of the ES nodes:
curl localhost:9200/logstash-2014.10.07/_search?pretty
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/
On Tue, Oct 7, 2014 at 6:40 PM, David Lang <david>@lang.hm> wrote:
> That looks like there is an error on the ES side of things, check the logs
> there. (the "no local error logger defined" line)
>
> David Lang
>
>
> On Tue, 7 Oct 2014, SjirBagmeijer wrote:
>
> Thank you for all the help I got much further now, it seems it was indeed
>> that I was supposed to use $programname instead of the $Tag, now it
looks
>> like it at-least connects to my elasticsearch according to the debug
>> information:
>>
>> 7794.801098530:7f38231c4700: omelasticsearch: beginTransaction
>> 7794.801103695:7f38231c4700: Action 0x7f382d70c650 transitioned to
state:
>> itx
>> 7794.801108636:7f38231c4700: entering actionCalldoAction(), state: itx
>> 7794.801171315:7f38231c4700: omelasticsearch: using REST URL: '
>> http://loghost.ulyaoth.net:9200/logstash-2014.10.07/events?'
>> 7794.801772371:7f38231c4700: omelasticsearch: pData replyLen = '108'
>> 7794.801782872:7f38231c4700: omelasticsearch: pData reply:
>> '{"_index":"logstash-2014.10.07","_type":"events","_id":"
>> BHzZDcXTTjqim0mLsglobA","_version":1,"created":true}'
>> 7794.801792573:7f38231c4700: omelasticsearch: no local error logger
>> defined - ignoring ES error information
>> 7794.801797633:7f38231c4700: omelasticsearch: result doAction: 0
>> (bulkmode 0)
>> 7794.801801664:7f38231c4700: Action 0x7f382d70c650 transitioned to
state:
>> rdy
>> 7794.801805504:7f38231c4700: action 0x7f382d70c650 call returned 0
>>
>> I cannot find it yet in Kibana somehow but I think I will be able to
>> figure that out :).
>>
>> -----Original Message-----
>> From: "Radu Gheorghe"<[email protected]>
>> To: "rsyslog-users"<[email protected]>;
>> Cc:
>> Sent: 2014-10-07 (Tue) 22:15:31
>> Subject: Re: [rsyslog] json files directly to ES
>>
>> Two more points from me that will hopefully help:
>> - if you're not sure where something breaks, try to isolate the
problem by
>> reducing the config to the bare minimum and building up on it once it
>> works. For example, I wouldn't bother with rulesets if no logs can get
to
>> ES in the first place. Just make sure you get your messages in and
through
>> to ES. If a minimal config fails, usually running rsyslog -dn like
David
>> advised should reveal the issue (for example, give you any exceptions
ES
>> generates or libcurl errors)
>> - if you're sure logs are already JSON, I wouldn't bother parsing
them. I
>> would just use templates to use the JSON as it is and eventually
enrich it
>> with new properties. Take a look here (scroll down to the last section)
>> for
>> an example:
>> http://wiki.rsyslog.com/index.php/Queues_on_v6_with_omelasticsearch
>>
>> It's outdated (uses old config format for most snippets) but it should
>> give
>> you some clues.
>>
>> Best regards,
>> Radu
>>
>> --
>> Performance Monitoring * Log Analytics * Search Analytics
>> Solr & Elasticsearch Support * http://sematext.com/
>>
>> On Tue, Oct 7, 2014 at 3:33 PM, David Lang <david>@lang.hm>
>> wrote:
>>
>> > you aren't showing us the entire config, so some of this is
>> guesswork.
>> >
>> > try logging the data with the format RSYSLOG_DebugFormat to see
what
>> > values are in each variable, the most common problem for things
>> going to
>> > the wrong place is that the variables you are testing don't have
the
>> value
>> > you expect.
>> >
>> > I think you need to test $programname not $Tag (Tag is what you
set
>> in the
>> > infile input, but it gets put in the $programname variable)
>> >
>> > Beyond that, I would look at the eleasticsearch logs to see if
it's
>> > complaining when you try to deliver the log.
>> >
>> > You can also start rsyslog in debug mode (-dn) to see all the gory
>> details
>> > of what it's doing, you should see it attempting to deliver the
log
>> and any
>> > error it gets back.
>> >
>> > David Lang
>> >
>> >
>> > On Tue, 7 Oct 2014, SjirBagmeijer wrote:
>> >
>> > Hello,
>> >>
>> >> I have a small question I am trying to make rsyslog to sent
log
>> files
>> >> that are already formatted in json directly into my
>> elasticsearch but I
>> >> have some trouble to get this to work is there someone that
>> could perhaps
>> >> see where I am going wrong with my config?
>> >>
>> >> Example of a log file content:
>> >> { "@timestamp": "2014-10-02T13:55:31+02:00", "message":
>> "127.0.0.1 - -
>> >> [02/Oct/2014:13:55:31 +0200] \"GET /_status HTTP/1.1\" 401 38
>> \"-\"
>> >> \"curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7
NSS/3.15.3
>> >> zlib/1.2.3 libidn/1.18 libssh2/1.4.2\"", "tags": ["nginx"],
>> "clientip":
>> >> "127.0.0.1", "remote_user": "-", "contenttype": "text/html;
>> charset=utf-8",
>> >> "bytes": 38, "duration": "0.012", "status": "401", "request":
>> "GET /_status
>> >> HTTP/1.1", "method": "GET", "referrer": "-", "useragent":
>> "curl/7.19.7
>> >> (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.15.3 zlib/1.2.3
>> libidn/1.18
>> >> libssh2/1.4.2" }
>> >>
>> >> Here below some examples that I tried to make by combining
>> examples from
>> >> you website:
>> >> http://www.rsyslog.com/tag/guides-for-rsyslog/
>> >> http://www.rsyslog.com/tag/elasticsearch/
>> >> http://www.rsyslog.com/doc/rsyslog_conf_filter.html
>> >>
>> >> I tried the following:
>> >> module(load="imfile" PollingInterval="10")
>> >> module(load="omelasticsearch")
>> >>
>> >> template(name="logstash-index"
>> >> type="list") {
>> >> constant(value="logstash-")
>> >> property(name="timereported" dateFormat="rfc3339"
>> position.from="1"
>> >> position.to="4")
>> >> constant(value=".")
>> >> property(name="timereported" dateFormat="rfc3339"
>> position.from="6"
>> >> position.to="7")
>> >> constant(value=".")
>> >> property(name="timereported" dateFormat="rfc3339"
>> position.from="9"
>> >> position.to="10")
>> >> }
>> >>
>> >> template(name="jsonULY" type="list") {
>> >> property(name="$!all-json")
>> >> }
>> >>
>> >> ruleset(name="logstash"){
>> >> action(type="omelasticsearch"
>> >> server="loghost.ulyaoth.net"
>> >> serverport="9200"
>> >> searchIndex="logstash-index"
>> >> dynSearchIndex="on"
>> >> template="jsonULY")
>> >> stop
>> >> }
>> >>
>> >>
>> >> input(type="imfile"
>> >> File="/var/log/nginx/access.json"
>> >> Tag="accessnginx"
>> >> StateFile="/var/spool/rsyslog/accessnginx"
>> >> ruleset="logstash")
>> >>
>> >> This seems to do nothing at all somehow, then I also tried the
>> following:
>> >> template(name="jsonULY" type="list") {
>> >> property(name="$!all-json")
>> >> }
>> >>
>> >> input(type="imfile"
>> >> File="/var/log/nginx/access.json"
>> >> Tag="nginxaccess"
>> >> StateFile="/var/spool/rsyslog/nginxaccess")
>> >>
>> >> if $Tag == 'nginxaccess' then {
>> >>
>> >> action(type="omelasticsearch"
>> >> server="logstash.ulyaoth.net"
>> >> serverport="9200"
>> >> searchIndex="logstash-index"
>> >> dynSearchIndex="on"
>> >> template="jsonULY")
>> >> stop
>> >> }
>> >>
>> >> And multiple other ways, it seems the input works but most of
>> the time it
>> >> is going directly to my /var/log/messages instead of going to
my
>> ES. I also
>> >> double checked that my ports are open.
>> >>
>> >> Is there someone that or can see what I do wrong and give me a
>> hint, or
>> >> perhaps a link to some example where people sent already
>> existing json
>> >> files directly to ES with rsyslog?
>> >>
>> >> Thank you so much in advanced.
>> >> _______________________________________________
>> >> rsyslog mailing list
>> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> >> http://www.rsyslog.com/professional-services/
>> >> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
by
>> a myriad
>> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
POST
>> if you
>> >> DON'T LIKE THAT.
>> >>
>> >> _______________________________________________
>> > rsyslog mailing list
>> > http://lists.adiscon.net/mailman/listinfo/rsyslog
>> > http://www.rsyslog.com/professional-services/
>> > What's up with rsyslog? Follow https://twitter.com/rgerhards
>> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>> myriad
>> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
>> you
>> > DON'T LIKE THAT.
>> >
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>>
>>
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>> DON'T LIKE THAT.
>>
>> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
