Hi, I'm not sure where Logstash fits in this picture - I thought you'd get JSONs from a file and send them to Elasticsearch.
ES version 1.1.1 should be OK, I'm not sure which rsyslog version you're on but all recent ones should work fine. You seem to have stuff in ES, but all the documents are empty (or at least _source is), so you either: - have a mapping where you don't include anything in _source. Less likely, you would have probably remembered that. You can check by running curl localhost:9200/logstash-2014.10.07/_mapping?pretty - rsyslog sends empty logs to ES. You can confirm this by writing the same logs to a file with the same template you use for ES. If logs are indeed empty, follow David's advice of using RSYSLOG_DebugFormat to see what rsyslog sees in each variable. Maybe things are parsed incorrectly Best regards, Radu -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Wed, Oct 8, 2014 at 8:47 AM, SjirBagmeijer <[email protected]> wrote: > Hello and thank you once more for the responses, so I have been testing > but it seems whatever I try it indeed does not somehow save the log in > ElasticSearch even it does say it did. > > I looked at ElasticSearch but there is no errors in the logs or any > strange behavior apart from this. > > The output from the command below is as following: > [loghost ~]# curl localhost:9200/logstash-2014.10.07/_search?pretty > { > "took" : 78, > "timed_out" : false, > "_shards" : { > "total" : 5, > "successful" : 5, > "failed" : 0 > }, > "hits" : { > "total" : 8232067, > "max_score" : 1.0, > "hits" : [ { > "_index" : "logstash-2014.10.07", > "_type" : "events", > "_id" : "NWKSiq5NTvysGkVZ9OR8XA", > "_score" : 1.0, "_source" : {} > }, { > "_index" : "logstash-2014.10.07", > "_type" : "events", > "_id" : "-5wf5CK_R5iCe1RF0zzsPg", > "_score" : 1.0, "_source" : {} > }, { > "_index" : "logstash-2014.10.07", > "_type" : "events", > "_id" : "LB8gFrTaRGyI5YreIPBS9w", > "_score" : 1.0, "_source" : {} > }, { > "_index" : "logstash-2014.10.07", > "_type" : "events", > "_id" : "aCck3E1GTqeanVAeHGGDsg", > "_score" : 1.0, "_source" : {} > }, { > "_index" : "logstash-2014.10.07", > "_type" : "events", > "_id" : "uCAMM4TGRD205AEMZMWtTQ", > "_score" : 1.0, "_source" : {} > }, { > "_index" : "logstash-2014.10.07", > "_type" : "events", > "_id" : "5JzfuFyXRiCxhhj73A249w", > "_score" : 1.0, "_source" : {} > }, { > "_index" : "logstash-2014.10.07", > "_type" : "events", > "_id" : "zNHmX6udT5GB7z5H2qD5bw", > "_score" : 1.0, "_source" : {} > }, { > "_index" : "logstash-2014.10.07", > "_type" : "events", > "_id" : "x_-nDe19SCCIuOKsd92CWw", > "_score" : 1.0, "_source" : {} > }, { > "_index" : "logstash-2014.10.07", > "_type" : "events", > "_id" : "s9wg8HhnRA6XSTf2kRMc1A", > "_score" : 1.0, "_source" : {} > }, { > "_index" : "logstash-2014.10.07", > "_type" : "events", > "_id" : "TGDNq7udT3uU5r-W6uI9TQ", > "_score" : 1.0, "_source" : {} > } ] > } > } > > It looks to me the logs somehow is not saved, I tried use some other > rsyslog configuration by shipping the logs to a tcp port and then let > logstash catch it this works without issues by doing this: > if $programname == 'default-nginx-accesslog' then @loghost:5544 > if $programname == 'default-nginx-accesslog' then ~ > > But this way it seems I have to grok all the logs correctly, any idea what > else I could look? Is there perhaps a version requirement of ElasticSearch > in order to sent it directly to it? > I currently use: (rpms on rhel7) > elasticsearch-1.1.1-1 > logstash-1.4.1-1 > > Thanks again, > Sjir Bagmeijer > > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
