Re: How to increase memory
Hi, it's 1.3.2
On Monday, August 18, 2014 5:49:03 PM UTC-4, Jörg Prante wrote:
>
> What version of ES do you use?
>
> Jörg
>
>
> On Mon, Aug 18, 2014 at 9:42 PM, rookie7799 > wrote:
>
>> Hello there,
>>
>> We are having the same exact problem with a really resource hungry query:
>> 5 nodes with 16GB ES_HEAP_SIZE
>> 1.2 Billion records inside 1 index with 5 shards
>>
>> Whenever we start running an aggregate query the whole cluster breaks and
>> disconnects. Why can't it just not return results and simple give and error
>> without actually killing the entire cluster?
>>
>> Cheers!
>>
>>
>> On Saturday, February 9, 2013 1:05:54 PM UTC-5, Igor Motov wrote:
>>>
>>> ES_HEAP_SIZE ES_MAX_MEM ES_MIN_MEM are environment variables. They need
>>> to be specified on the command line. For example:
>>>
>>> ES_HEAP_SIZE=4g bin/elasticsearch -f
>>>
>>> To get JVM stats, you need to set jvm=true on stats request:
>>>
>>> curl -XGET 'http://localhost:9200/_cluster/nodes/stats?jvm=true&;
>>> pretty=true'
>>>
>>> To understand how much memory you need, give it as much as you can, put
>>> some load and monitor "jvm.mem.heap_used" in the output of the stats
>>> command above. If this number ever goes and stays above 90%
>>> of available heap it's typically a good indicator that you need more.
>>>
>>> There is a small Russian elasticsearch forum -
>>> https://groups.google.com/forum/?fromgroups=#!forum/elasticsearch-ru
>>>
>>> On Saturday, February 9, 2013 12:57:04 PM UTC-5, Николай Измайлов wrote:
>>>>
>>>> In continuation of the topic https://github.com/
>>>> elasticsearch/elasticsearch/issues/2636#issuecomment-13332877
>>>>
>>>> in continuation of the topic https://github.com/
>>>> elasticsearch/elasticsearch/issues/2636#issuecomment-13332877
>>>> On the page http://www.elasticsearch.org/guide/reference/setup/
>>>> installation.html it is said that it is necessary to increase
>>>> ES_HEAP_SIZE ES_MAX_MEM ES_MIN_MEM, but I have not found this
>>>> configuration
>>>> then /etc/elasticsearch/elasticsearch.yml. Here's my cluster
>>>>
>>>> {
>>>> "cluster_name" : "elasticsearch",
>>>> "nodes" : {
>>>> "VPjABUm-REmy24NQ_AkXDQ" : {
>>>> "timestamp" : 1360432148849,
>>>> "name" : "Sin",
>>>> "transport_address" : "inet[/ip:9300]",
>>>> "hostname" : "Ubuntu-1204-precise-64-minimal",
>>>> "indices" : {
>>>> "store" : {
>>>> "size" : "34.6gb",
>>>> "size_in_bytes" : 37221752556,
>>>> "throttle_time" : "0s",
>>>> "throttle_time_in_millis" : 0
>>>> },
>>>> "docs" : {
>>>> "count" : 58480,
>>>> "deleted" : 4759
>>>> },
>>>> "indexing" : {
>>>> "index_total" : 20,
>>>> "index_time" : "1.7s",
>>>> "index_time_in_millis" : 1748,
>>>> "index_current" : 0,
>>>> "delete_total" : 0,
>>>> "delete_time" : "0s",
>>>> "delete_time_in_millis" : 0,
>>>> "delete_current" : 0
>>>> },
>>>> "get" : {
>>>> "total" : 2,
>>>> "time" : "5ms",
>>>> "time_in_millis" : 5,
>>>> "exists_total" : 0,
>>>> "exists_time" : "0s",
>>>> "exists_time_in_millis" : 0,
>>>> "missing_total" : 2,
>>>> "missing_time" : "5ms",
>>>> "missing_time_in_millis" : 5,
>>>> "current" : 0
>>>> },
>>>> "search" : {
>>>> "query_total" : 1726375,
&g
Re: How to increase memory
Hello there,
We are having the same exact problem with a really resource hungry query:
5 nodes with 16GB ES_HEAP_SIZE
1.2 Billion records inside 1 index with 5 shards
Whenever we start running an aggregate query the whole cluster breaks and
disconnects. Why can't it just not return results and simple give and error
without actually killing the entire cluster?
Cheers!
On Saturday, February 9, 2013 1:05:54 PM UTC-5, Igor Motov wrote:
>
> ES_HEAP_SIZE ES_MAX_MEM ES_MIN_MEM are environment variables. They need to
> be specified on the command line. For example:
>
> ES_HEAP_SIZE=4g bin/elasticsearch -f
>
> To get JVM stats, you need to set jvm=true on stats request:
>
> curl -XGET '
> http://localhost:9200/_cluster/nodes/stats?jvm=true&pretty=true'
>
> To understand how much memory you need, give it as much as you can, put
> some load and monitor "jvm.mem.heap_used" in the output of the stats
> command above. If this number ever goes and stays above 90%
> of available heap it's typically a good indicator that you need more.
>
> There is a small Russian elasticsearch forum -
> https://groups.google.com/forum/?fromgroups=#!forum/elasticsearch-ru
>
> On Saturday, February 9, 2013 12:57:04 PM UTC-5, Николай Измайлов wrote:
>>
>> In continuation of the topic
>> https://github.com/elasticsearch/elasticsearch/issues/2636#issuecomment-13332877
>>
>> in continuation of the topic
>> https://github.com/elasticsearch/elasticsearch/issues/2636#issuecomment-13332877
>> On the page
>> http://www.elasticsearch.org/guide/reference/setup/installation.html it
>> is said that it is necessary to increase ES_HEAP_SIZE ES_MAX_MEM
>> ES_MIN_MEM, but I have not found this configuration then
>> /etc/elasticsearch/elasticsearch.yml. Here's my cluster
>>
>> {
>> "cluster_name" : "elasticsearch",
>> "nodes" : {
>> "VPjABUm-REmy24NQ_AkXDQ" : {
>> "timestamp" : 1360432148849,
>> "name" : "Sin",
>> "transport_address" : "inet[/ip:9300]",
>> "hostname" : "Ubuntu-1204-precise-64-minimal",
>> "indices" : {
>> "store" : {
>> "size" : "34.6gb",
>> "size_in_bytes" : 37221752556,
>> "throttle_time" : "0s",
>> "throttle_time_in_millis" : 0
>> },
>> "docs" : {
>> "count" : 58480,
>> "deleted" : 4759
>> },
>> "indexing" : {
>> "index_total" : 20,
>> "index_time" : "1.7s",
>> "index_time_in_millis" : 1748,
>> "index_current" : 0,
>> "delete_total" : 0,
>> "delete_time" : "0s",
>> "delete_time_in_millis" : 0,
>> "delete_current" : 0
>> },
>> "get" : {
>> "total" : 2,
>> "time" : "5ms",
>> "time_in_millis" : 5,
>> "exists_total" : 0,
>> "exists_time" : "0s",
>> "exists_time_in_millis" : 0,
>> "missing_total" : 2,
>> "missing_time" : "5ms",
>> "missing_time_in_millis" : 5,
>> "current" : 0
>> },
>> "search" : {
>> "query_total" : 1726375,
>> "query_time" : "7.7m",
>> "query_time_in_millis" : 462631,
>> "query_current" : 0,
>> "fetch_total" : 61663,
>> "fetch_time" : "20.9s",
>> "fetch_time_in_millis" : 20955,
>> "fetch_current" : 0
>> },
>> "cache" : {
>> "field_evictions" : 0,
>> "field_size" : "0b",
>> "field_size_in_bytes" : 0,
>> "filter_count" : 5896,
>> "filter_evictions" : 0,
>> "filter_size" : "511.6kb",
>> "filter_size_in_bytes" : 523944,
>> "bloom_size" : "22.1kb",
>> "bloom_size_in_bytes" : 22640,
>> "id_cache_size" : "0b",
>> "id_cache_size_in_bytes" : 0
>> },
>> "merges" : {
>> "current" : 0,
>> "current_docs" : 0,
>> "current_size" : "0b",
>> "current_size_in_bytes" : 0,
>> "total" : 0,
>> "total_time" : "0s",
>> "total_time_in_millis" : 0,
>> "total_docs" : 0,
>> "total_size" : "0b",
>> "total_size_in_bytes" : 0
>> },
>> "refresh" : {
>> "total" : 15,
>> "total_time" : "143ms",
>> "total_time_in_millis" : 143
>> },
>> "flush" : {
>> "total" : 25,
>> "total_time" : "3.2s",
>> "total_time_in_millis" : 3205
>> }
>> }
>> }
>> }
>> }
>>
>>
>> As understand how much I need to allocate memory for elasticsearch and in
>> General the description for each of the parameters.
>>
>> there is a Russian community ?
>>
>>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussio
Re: Java api TransportClient question
You're right, it was a memory issue at the end. However, it's strange that
java api never complained about it.
Thank you for reply!
On Friday, August 8, 2014 4:41:04 PM UTC-4, Jörg Prante wrote:
>
> In the response, you can see that not all shards did respond. There is
> something wrong with the shards. Regarding the very high response time, I
> assume they got tight resources like memory or something at least they
> dropped from the overall search results without timeout. Maybe there is
> something in the server node logs.
>
> Jörg
>
>
> On Fri, Aug 8, 2014 at 9:20 PM, rookie7799 > wrote:
>
>> Hello there,
>>
>> Setup:
>> 4 node cluster
>> 1 index (tweets) (1.1 billion indexed, 5 shards)
>> using java api
>>
>> java client:
>> Settings settings = ImmutableSettings.settingsBuilder()
>> .put("cluster.name", "tweets").build();
>>
>> Client client = new TransportClient(settings)
>> .addTransportAddress(new InetSocketTransportAddress("#",
>> #))
>> .addTransportAddress(new InetSocketTransportAddress("#",
>> #))
>> .addTransportAddress(new InetSocketTransportAddress("#",
>> #))
>> .addTransportAddress(new InetSocketTransportAddress("#",
>> #));
>>
>> ...
>> ...
>> ...
>> QueryBuilder qb =
>> QueryBuilders.rangeQuery("tweet_date").gte(dt1).lte(dt2);
>> SearchRequestBuilder srb = client.prepareSearch("tweets");
>> srb.setTypes("tweet");
>> srb.setSearchType(SearchType.DFS_QUERY_THEN_FETCH);
>> SearchResponse sr =
>> srb.setQuery(qb).setSize(0).srb.execute().actionGet()
>>
>>
>>
>> For some reason when I execute a "range date" query when results come
>> back only partial shards are "successful", eg:
>>
>> "took" : 27119,
>> "timed_out" : false,
>> "_shards" : {
>> "total" : 5,
>> "successful" : 1,
>> "failed" : 0
>> },
>> "hits" : {
>> "total" : 224795578,
>> "max_score" : 0.0,
>> "hits" : [ ]
>> },
>>
>> sometimes it's:
>>
>> "took" : 31076,
>> "timed_out" : false,
>> "_shards" : {
>> "total" : 5,
>> "successful" : 3,
>> "failed" : 0
>> },
>> "hits" : {
>> "total" : 674358780,
>> "max_score" : 0.0,
>> "hits" : [ ]
>> },
>>
>> is there a way to make sure all shards return all the information? Or am
>> I doing something wrong?
>>
>> Thank you !
>>
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/651cf463-fbd1-4fa7-ac74-b9ca1520fcb5%40googlegroups.com
>>
>> <https://groups.google.com/d/msgid/elasticsearch/651cf463-fbd1-4fa7-ac74-b9ca1520fcb5%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/e9315667-cf40-4a3f-a814-d08922caf661%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Java api TransportClient question
Hello there,
Setup:
4 node cluster
1 index (tweets) (1.1 billion indexed, 5 shards)
using java api
java client:
Settings settings = ImmutableSettings.settingsBuilder()
.put("cluster.name", "tweets").build();
Client client = new TransportClient(settings)
.addTransportAddress(new InetSocketTransportAddress("#", #))
.addTransportAddress(new InetSocketTransportAddress("#", #))
.addTransportAddress(new InetSocketTransportAddress("#", #))
.addTransportAddress(new InetSocketTransportAddress("#",
#));
...
...
...
QueryBuilder qb = QueryBuilders.rangeQuery("tweet_date").gte(dt1).lte(dt2);
SearchRequestBuilder srb = client.prepareSearch("tweets");
srb.setTypes("tweet");
srb.setSearchType(SearchType.DFS_QUERY_THEN_FETCH);
SearchResponse sr =
srb.setQuery(qb).setSize(0).srb.execute().actionGet()
For some reason when I execute a "range date" query when results come back
only partial shards are "successful", eg:
"took" : 27119,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 1,
"failed" : 0
},
"hits" : {
"total" : 224795578,
"max_score" : 0.0,
"hits" : [ ]
},
sometimes it's:
"took" : 31076,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 3,
"failed" : 0
},
"hits" : {
"total" : 674358780,
"max_score" : 0.0,
"hits" : [ ]
},
is there a way to make sure all shards return all the information? Or am I
doing something wrong?
Thank you !
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/651cf463-fbd1-4fa7-ac74-b9ca1520fcb5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
