Re: Elasticsearch Memory issue

[AlexK]

Hi Jorg,

Thanks for the answer.

The idea behind the restriction of a single machine was for instance to 
install ELK on a machine and perform fast indexing and review of a set of 
log. What I got wrong is that the log size can be important (hundreds of 
Gb) so this architecture will not work, according to the answers 
above...(goal was to replace Splunk in a similar set up)

Thanks,
Alex.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/94ccef51-1cd9-4210-9f6c-d622108633df%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elasticsearch Memory issue

[joergpra...@gmail.com]

With ES on a single machine, "tuning" does not cure the symptoms in the
long run. ES was designed to scale out on many nodes, so the simplest path
is to add nodes.

In a restricted environment, you could try to disable features that consume
a fairly amount of resources: disable _source and _all field, bloom codec
filter cache, reduce complex analyzers in the mapping, reduce number of
shards, reduce filter caches ...

Jörg


On Mon, Jun 30, 2014 at 5:04 PM, AlexK  wrote:

> Hi,
>
> I am using ES version 1.1.1 on a *single node* with the below settings:
>  - 2 shards per indices;
>  - currently 395 indices;
>  - currently *428GB*;
>  - *18GB* Heap commited on a 24 GB machine;
>
> I used it only for indexing now, because I noted some memory issues when
> performing search: elasticsearch became unresponsive...
> My requirement curently is to  use ElasticSearch only on a single machine
> configuration.
>
> I already tried to tune the JVM (Xmn to 2gb, noting Full Garbarge
> Collector very often) whithout success.
>
> Does anyone have some advices for me ?
>
> 1. Is the only way is to increase the heap (do you have a ratio RAM / data
> size) ?
> 2. Does adding additional nodes on the same machine will help, why ?
> 3. Final question, I cannot see why in the indexing part so much of the
> RAM is used, if anybody can explain ?
>
> Thanks a lot for your answers.
> Alex.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/26d47903-34a2-4c5b-8ffd-3f20ab2bfc20%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHB_jCLt7%3DR4H4O_wy-sAhGx90TrpW%3Du8AHXROdemNyYA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elasticsearch Memory issue

[AlexK]

Setting up the JVM memory to 50% (12G) did not ease the problem as I 
noticed GC collection up to 3min :)
Will really need to add a bunch of RAM to my machine..

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/874f127a-639a-4416-bdec-d50d77920f45%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elasticsearch Memory issue

[smonasco]

My understanding was that ES 1.1 was using memory mapped files and so the field 
cache would not be part of the heap.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/308f68ba-bd7a-4775-ae1d-90088445d103%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elasticsearch Memory issue

[AlexK]

Thanks both of you for the answer.

I have just decrease the Java Jvm memory to 50% (12G), I will see if that 
helps.

@Jilles:
- I am using the defaut Logstash template and I things that by default the 
_all field is disable...Ah no that not the case :( I will correct this 
settings but why by default this field is enabled ?
- I will set a limit for the field cache. But regarding the circuit 
breaker, what are these ?
- I totally agree that it would be good to have a deep tuning guide on ES :)

Thanks again,
Alex

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/3894e4b5-eb8a-4c0d-90ff-75eb884743f9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elasticsearch Memory issue

[Jilles van Gurp]

You should tweak cache sizes. At least the field data cache needs to be 
restricted (unbounded by default). Also, ensuring the various circuit 
breakers are turned on will help. Another tip is to disable the _all field 
if you don't need it. 

All this should reduce the amount of memory ES uses and make it less likely 
your cluster becomes unavailable. We use Elasticsearch with Kibana in our 
production setup. Things definitely do not fail gracefully if you run short 
of memory; so you need to prevent that situation. I've had a completely 
unresponsive cluster on two occasions. With the current settings, it has 
been running stable for several weeks now.

I've learned a few of these things the hard way. I think a ES tuning guide 
for non experts is desperately needed. The out of the box experience is not 
really appropriate for any serious production environment. But then, you 
wouldn't run mysql with default settings in production either. In my 
experience, you currently need to piece together bits of good advice spread 
through the documentation and various forum posts. If you have an untuned 
Elasticsearch in production there are several failure scenarios that are 
likely to result in unavailability and data loss. Especially if you are 
using ELK with lots of log data, you need to tune or you basically will 
have a dead cluster in no time due to OOMs. 

Jilles

On Monday, June 30, 2014 5:04:12 PM UTC+2, AlexK wrote:
>
> Hi,
>
> I am using ES version 1.1.1 on a *single node* with the below settings:
>  - 2 shards per indices;
>  - currently 395 indices;
>  - currently *428GB*;
>  - *18GB* Heap commited on a 24 GB machine;
>
> I used it only for indexing now, because I noted some memory issues when 
> performing search: elasticsearch became unresponsive...
> My requirement curently is to  use ElasticSearch only on a single machine 
> configuration.
>
> I already tried to tune the JVM (Xmn to 2gb, noting Full Garbarge 
> Collector very often) whithout success.
>
> Does anyone have some advices for me ?
>
> 1. Is the only way is to increase the heap (do you have a ratio RAM / data 
> size) ? 
> 2. Does adding additional nodes on the same machine will help, why ?
> 3. Final question, I cannot see why in the indexing part so much of the 
> RAM is used, if anybody can explain ?
>
> Thanks a lot for your answers.
> Alex.
>  
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/5e79bff1-409e-4389-8be3-69590f3ffb56%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elasticsearch Memory issue

[Mark Walkom]

When it comes to capacity the answer is, it depends.
Given you're at around 430GB on a single node now, I'd add another node and
then see how things look at around the 8-900GB mark (spread across both).


Another clarification; The recommended operating procedure is to use half
your system RAM for heap, leaving the other half for OS caching, which
increases performance of ES. In your case under the best possible
circumstances, you should only really be using 12GB of your total 24GB.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com


On 1 July 2014 17:30, AlexK  wrote:

> Hi Mark,
>
> Thanks for your quick answer.
> I cannot increase the RAM for ES, as I am already using 75% of the ram for
> the JVM.
> I will take a look at disabling the bloom filter cache to see if that
> change anything.
>
> Regarding the option of adding more nodes:
>  - Do you have an idea of how many nodes are required to sustain that
> quantity of data (428Gb), I will say 3 nodes one for each indice and an
> extra node for replica, do you agree ?
>  - In the hypothesis that I will not use in a first time replicas, do you
> have an idea about the quantity of RAM needed for the two nodes (continuous
> indexing and few search requests) ?
>
> Again, thanks for your answers.
> Alex.
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/e4a86ea4-008d-4caf-b101-94d8278cad03%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEM624Z5dt7GP%3DnhfijLBUc_nuaswMQ6gzGgG0FuM4i4BfSMoA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elasticsearch Memory issue

[AlexK]

Hi Mark,

Thanks for your quick answer.
I cannot increase the RAM for ES, as I am already using 75% of the ram for 
the JVM.
I will take a look at disabling the bloom filter cache to see if that 
change anything.

Regarding the option of adding more nodes:
 - Do you have an idea of how many nodes are required to sustain that 
quantity of data (428Gb), I will say 3 nodes one for each indice and an 
extra node for replica, do you agree ?
 - In the hypothesis that I will not use in a first time replicas, do you 
have an idea about the quantity of RAM needed for the two nodes (continuous 
indexing and few search requests) ?

Again, thanks for your answers.
Alex.

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/e4a86ea4-008d-4caf-b101-94d8278cad03%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Elasticsearch Memory issue

[Mark Walkom]

You are more than likely reaching the limits of the node.
Your options are to delete data, add more RAM (you should have 50% of
system for heap), close some old indexes or add nodes. Adding more nodes
spreads the shards of your indexes across the cluster which is essentially
spreading the load.

You could try disabling bloom filter cache as this will reduce your memory
usage a little, take a look Elasticsearch Curator which can do this for you.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com


On 1 July 2014 01:04, AlexK  wrote:

> Hi,
>
> I am using ES version 1.1.1 on a *single node* with the below settings:
>  - 2 shards per indices;
>  - currently 395 indices;
>  - currently *428GB*;
>  - *18GB* Heap commited on a 24 GB machine;
>
> I used it only for indexing now, because I noted some memory issues when
> performing search: elasticsearch became unresponsive...
> My requirement curently is to  use ElasticSearch only on a single machine
> configuration.
>
> I already tried to tune the JVM (Xmn to 2gb, noting Full Garbarge
> Collector very often) whithout success.
>
> Does anyone have some advices for me ?
>
> 1. Is the only way is to increase the heap (do you have a ratio RAM / data
> size) ?
> 2. Does adding additional nodes on the same machine will help, why ?
> 3. Final question, I cannot see why in the indexing part so much of the
> RAM is used, if anybody can explain ?
>
> Thanks a lot for your answers.
> Alex.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/26d47903-34a2-4c5b-8ffd-3f20ab2bfc20%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAEM624YSdjUQnQJ_Wwu82iHxT5m%3DZVDRR_pskCXQ2RPXmLHRxQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Elasticsearch Memory issue

[AlexK]

Hi,

I am using ES version 1.1.1 on a *single node* with the below settings:
 - 2 shards per indices;
 - currently 395 indices;
 - currently *428GB*;
 - *18GB* Heap commited on a 24 GB machine;

I used it only for indexing now, because I noted some memory issues when 
performing search: elasticsearch became unresponsive...
My requirement curently is to  use ElasticSearch only on a single machine 
configuration.

I already tried to tune the JVM (Xmn to 2gb, noting Full Garbarge Collector 
very often) whithout success.

Does anyone have some advices for me ?

1. Is the only way is to increase the heap (do you have a ratio RAM / data 
size) ? 
2. Does adding additional nodes on the same machine will help, why ?
3. Final question, I cannot see why in the indexing part so much of the RAM 
is used, if anybody can explain ?

Thanks a lot for your answers.
Alex.
 

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/26d47903-34a2-4c5b-8ffd-3f20ab2bfc20%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.