Re: heap size/filter cache
Thank you for answer. We had lot's of cache evictions so we increased cache size. Should I use custom cache keys and try to make it smaller? What cache keys are generated by default? Is it full filter query dsl string? 2014 m. lapkritis 29 d., šeštadienis 13:01:59 UTC+2, Adrien Grand rašė: When you cache filters, there are two things that can take memory: the cached bitsets and the filter itself (which is used as a key). And while we have APIs to estimate the memory usage for bitsets, we do not have anything for filters unfortunately. So if you happen to have lots of small segments on your machine or if you use filters that can have a high memory footprint (like terms filters with lots of terms) then the memory usage of your filter cache might be underestimated because of the filters which are not taken into account. By the way, 20% is a bit high for a filter cache size. Recent versions of elasticsearch have a default value of 10% ( http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-cache.html) which should already allow for caching lots of caches. On Fri, Nov 28, 2014 at 9:58 AM, Audrius Bugas audriu...@hometogo.com javascript: wrote: Hello, I have 8GB dedicated for heap, 20% heap is dedicated for filters cache. When clear filter cache from api it clears ~4GB memory from heap (way more then filters cache). Is it normal, or I have some problems with cached filters? We are using warmers, but this should not be a reason... -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com javascript:. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9467f35b-b48c-4a49-949d-250ea6ea8e5d%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/9467f35b-b48c-4a49-949d-250ea6ea8e5d%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- Adrien Grand -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/32533606-c3d8-48fe-8020-d2bef3ee42c7%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: heap size/filter cache
Cache keys are pretty much the input filter by default. Specifying your own keys can certainly help. On Dec 1, 2014 7:05 AM, Audrius Bugas bugas.audr...@gmail.com wrote: Thank you for answer. We had lot's of cache evictions so we increased cache size. Should I use custom cache keys and try to make it smaller? What cache keys are generated by default? Is it full filter query dsl string? 2014 m. lapkritis 29 d., šeštadienis 13:01:59 UTC+2, Adrien Grand rašė: When you cache filters, there are two things that can take memory: the cached bitsets and the filter itself (which is used as a key). And while we have APIs to estimate the memory usage for bitsets, we do not have anything for filters unfortunately. So if you happen to have lots of small segments on your machine or if you use filters that can have a high memory footprint (like terms filters with lots of terms) then the memory usage of your filter cache might be underestimated because of the filters which are not taken into account. By the way, 20% is a bit high for a filter cache size. Recent versions of elasticsearch have a default value of 10% (http://www.elasticsearch.org/ guide/en/elasticsearch/reference/current/index-modules-cache.html) which should already allow for caching lots of caches. On Fri, Nov 28, 2014 at 9:58 AM, Audrius Bugas audriu...@hometogo.com wrote: Hello, I have 8GB dedicated for heap, 20% heap is dedicated for filters cache. When clear filter cache from api it clears ~4GB memory from heap (way more then filters cache). Is it normal, or I have some problems with cached filters? We are using warmers, but this should not be a reason... -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/ msgid/elasticsearch/9467f35b-b48c-4a49-949d-250ea6ea8e5d% 40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/9467f35b-b48c-4a49-949d-250ea6ea8e5d%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- Adrien Grand -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/32533606-c3d8-48fe-8020-d2bef3ee42c7%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/32533606-c3d8-48fe-8020-d2bef3ee42c7%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPmjWd0s%3DS5kZbw_cug2LB1Lt%2BekOGGGzmGF_uAJRkW6yN5s%3Dg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: heap size/filter cache
When you cache filters, there are two things that can take memory: the cached bitsets and the filter itself (which is used as a key). And while we have APIs to estimate the memory usage for bitsets, we do not have anything for filters unfortunately. So if you happen to have lots of small segments on your machine or if you use filters that can have a high memory footprint (like terms filters with lots of terms) then the memory usage of your filter cache might be underestimated because of the filters which are not taken into account. By the way, 20% is a bit high for a filter cache size. Recent versions of elasticsearch have a default value of 10% ( http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-cache.html) which should already allow for caching lots of caches. On Fri, Nov 28, 2014 at 9:58 AM, Audrius Bugas audrius.bu...@hometogo.com wrote: Hello, I have 8GB dedicated for heap, 20% heap is dedicated for filters cache. When clear filter cache from api it clears ~4GB memory from heap (way more then filters cache). Is it normal, or I have some problems with cached filters? We are using warmers, but this should not be a reason... -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9467f35b-b48c-4a49-949d-250ea6ea8e5d%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/9467f35b-b48c-4a49-949d-250ea6ea8e5d%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- Adrien Grand -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6L7OB5Fxc364z6%3DMt-ecSmRpgsxu%3D0OR-vZ9ODjqRR0w%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
