Hi all, We've been running an ElasticSearch cluster of three nodes since last December. We're running them on Debian Wheezy. Due to the size of our network, we're getting about 600 messages/s (800 at peak times). Using logstash and daily indices, we're currently at about 3TB of data spread over 158 indices. We use 2 replicas, so all machines have all data.
I'm struggling to understand what expected working limits are for ElasticSearch in a centralized log situation and would really appreciate input from others. What we eventually try to do is provide about 13 months of searchable logs via Kibana, but we're mostly running into RAM constraints. Each node has 3TB of data, but the heap usage is almost constantly up at 27-29GB. We did have a problem with garbage collects earlier, which took so long that a node would drop from the cluster. To fix this, we switched to the G1 gc, which seems very well suited for this operation. The machines we're using have 12 cores, so the added CPU overhead is negligible (general usage of the cores is less than 30%). But I'm not enough of a Java dev to judge if this switch could be the cause of the constant high heap usage. We're currently at about 1GB RAM per 100GB of disk usage. My questions: 1- Is 1GB RAM usage per 100GB disk usage an expected usage pattern for an index heavy cluster? 2- Aside from closing indices, are there other ways of lowering this? 2.5- Should I worry about it? 3- Are we approaching this the wrong way and should we change our setup? 4- Would upgrading to 1.3.1 change the usage significantly, due to the fix in issue 6856 or is it unrelated? The numbers: 3 nodes with the following config: - 92GB of RAM - 12 cores with HT - 6x 3.6TB spinning disks (we're contemplating adding CacheCade, since the controller supports it) - We expose the disks to elasticsearch via 6 mount points - Debian Wheezy - Java 1.7.0_65 OpenJDK JRE with IcedTea 2.5.1 (Debian package with version 7u65-2.5.1-2~deb7u1) - ElasticSearch 1.2.1 from the ElasticSearch repositories Config snippets (leaving out ips and the like): bootstrap: mlockall: true cluster: name: logging routing: allocation: node_concurrent_recoveries: 4 discovery: zen: minimum_master_nodes: 2 ping: unicast: hosts: [snip] index: number_of_replicas: 2 number_of_shards: 6 indices: memory: index_buffer_size: 50% recovery: concurrent_streams: 5 max_bytes_per_sec: 100mb node: concurrent_recoveries: 6 name: stor1-stor path: data: - /srv/elasticsearch/a - /srv/elasticsearch/b - /srv/elasticsearch/c - /srv/elasticsearch/d - /srv/elasticsearch/f - /srv/elasticsearch/e Java options: -Xms30g -Xmx30g -Xss256k -Djava.awt.headless=true -XX:+UseG1GC -XX:+HeapDumpOnOutOfMemoryError Thanks for your help! Please let me know if you need more information -- Kind regards, Tim Stoop -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2846daa5-a75c-4d04-ab34-957984c9e05e%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.