3,000 events/sec Architecture

2014-03-04 Thread Eric Luellen
Hello, I've been working on a POC for Logstash/ElasticSearch/Kibana for about 2 months now and everything has worked out pretty good and we are ready to move it to production. Before building out the infrastructure, I want to make sure my shard/node/index setup is correct as that is the main

Re: 3,000 events/sec Architecture

2014-03-04 Thread Eric Luellen
to this :) -Zachary On Tuesday, March 4, 2014 9:11:59 AM UTC-6, Eric Luellen wrote: Hello, I've been working on a POC for Logstash/ElasticSearch/Kibana for about 2 months now and everything has worked out pretty good and we are ready to move it to production. Before building out the infrastructure, I want

Re: Index Mapping/Routing Help

2014-02-18 Thread Eric Luellen
Thanks for that information. When I'm looking in Kibana now, it's showing the correct type but it still shows the index as the original logstash-2014-02-18. Not sure why it isn't showing the unix-date index. If I look at ElasticSearch, I can see that it did create the new index I told it to

Index Mapping/Routing Help

2014-02-17 Thread Eric Luellen
Hello, Currently I have the following setup. Syslog -- Logstash -- ElasticSearch -- Kibana Logstash is creating a daily index /etc/elasticsearch/data/test-elasticsearch/nodes/0/indices/logstash-2014.02.04 and I'm viewing all of the logs through Kibana. We want to set up some user based

Elasticsearch Missing Data

2014-01-08 Thread Eric Luellen
Hello, I've had my elasticsearch instance running for about a week with no issues, but last night it stopped working. When I went to look in Kibana, it stops logging around 20:45 on 1/7/14. I then restarted the service on both both elasticsearch servers and it started logging again and back

ElasticSearch Index Wrong Date

2014-01-02 Thread Eric Luellen
Hello, I recently setup my elasticsearch instance and everything has been working fine. However, when I looked at Kibana today I saw that the logs stopped showing up as soon as 2014 hit. When looking at my data on the cluster, I see this: ls -altr data/my-cluster/nodes/0/indices/ total 44

Re: Unassigned Shards

2013-12-20 Thread Eric Luellen
: www.campaignmonitor.com On 20 December 2013 08:52, Eric Luellen eric.l...@gmail.com javascript: wrote: I think I made my situation even worse. I tried deleting the shards and starting over and now elasticsearch isn't even creating the /etc/elasticsearch/data/my-cluster/node folder

Re: Unassigned Shards

2013-12-20 Thread Eric Luellen
I got the initial issue fixed of me getting data back again. However I still don't understand how to fix the unassigned shards issue and how to properly restart elasticsearch without it complaining. On Friday, December 20, 2013 9:28:53 AM UTC-5, Eric Luellen wrote: Mark, I used the rpm

Help with Cluster

2013-12-17 Thread Eric Luellen
I am working on building out a small POC for Logstash and Elasticsearch. To start, I have a 2 server setup. - Server 1 - logstash1 - running java -jar logstash-1.2.2-flatjar.jar agent -f indexer.conf - This server is tailing logs from a syslog config file and then sending them