Eric, as an update, I hit OOM with a couple nodes in my cluster today w/
16gb ram for ES alone (each data node has 24gb ram) - I was running fine,
but then I had users kick off regular searches to watch performance, and my
indexing rates went from 35k/sec down to almost nothing (ran at a lesser
Apache Flume has the necessary pieces.
Otis
--
Performance Monitoring * Log Analytics * Search Analytics
Solr Elasticsearch Support * http://sematext.com/
On Wednesday, March 12, 2014 5:01:37 AM UTC-4, Jörg Prante wrote:
It would also be possible to write a custom Java syslog protocol socket
Yes, currently logstash is reading files that syslog-ng created. We already
had the syslog-ng architecture in place so just kept rolling with that.
On Tuesday, March 11, 2014 11:16:42 PM UTC-4, Otis Gospodnetic wrote:
Hi,
Is that Logstash instance reading files that are produces by
Hi,
Is that Logstash instance reading files that are produces by syslog-ng
servers? Maybe not but if yes, have you considered using Rsyslog with
omelasticsearch instead to simplify the architecture?
Otis
--
Performance Monitoring * Log Analytics * Search Analytics
Solr Elasticsearch
Hello,
I've been working on a POC for Logstash/ElasticSearch/Kibana for about 2
months now and everything has worked out pretty good and we are ready to
move it to production. Before building out the infrastructure, I want to
make sure my shard/node/index setup is correct as that is the main
Based on my experience, I think you may have an issue with OOM trying to
keep a month of logs with ~10gb ram / server.
Say, for instance, 5 indexes a day for 30 days = 150 indexes. How many
shards per index/replicas?
I ran some tests with 8GB assigned to my 20x ES data nodes, and after a ~7
Zach,
Thanks for the information. With my POC, I have 2 10 gig VMs and I'm
keeping 7 days of logs with no issues but that is a fairly large jump and I
could see where it may pose an issue.
As far as the 150 indexes, I'm not sure on the shards per index/replicas.
That is the part that I'm the
My initial suggestion would be to set your templates to 3 shards, 1
replica. With three data nodes, you'd have two shards per index, at 5
indexes/day, that's 10 shards per day per index per node. 3 nodes/10
shards per day/30 days is 900 shards. I don't know any 'cutoff' per se,
but 900 may
