Re: Elasticsearch swapping
Hi Alex,
Thank you for your comments.
You can see the curl command result (for security reason, ip addresses
replaced with ). For all 3 nodes, bootstrap.mlockall is true.
Also doesn't matter on which node I run the command.
{"ok":true,"cluster_name":"","nodes":{"rqptXnAMS7O-scqGJTIssQ":{"name":
".42:1","transport_address":"inet[bdm01-test/.42:9300]",
"hostname":"bdm01-test","version":"0.90.3","http_address":
"inet[/.42:9200]","settings":{"path.home":"/home/nli/opt/bdm","pidfile":
"/home/nli/opt/bdm/pid/.42:1/elasticsearch.pid","path.logs":
"/home/nli/opt/bdm/log/elasticsearch/.42:1","path.work":
"/tmp/elasticsearch/.42:1","max-open-files":"true","path.data":
"/home/nli/data/elasticsearchdata","config":
"/home/nli/opt/bdm/config/.42:1.yml",
"discovery.zen.ping.unicast.hosts.2":".44:9300",
"discovery.zen.ping.unicast.hosts.1":".43:9300",
"gateway.recovery_after_time":"5m","discovery.zen.ping.unicast.hosts.0":
".42:9300","indices.ttl.interval":"86400s","bootstrap.mlockall":"true",
"cluster.name":"","http.port":"9200","network.publish_host":".42",
"indices.memory.index_buffer_size":"20%",
"discovery.zen.minimum_master_nodes":"2","discovery.zen.fd.ping_timeout":
"60s","gateway.recovery_after_nodes":"3","gateway.expected_nodes":"3",
"discovery.zen.ping.timeout":"30s","indices.fielddata.cache.expire":"1m",
"discovery.zen.fd.ping_retries":"10","node.name":".42:1",
"indices.fielddata.cache.size":"10%","action.auto_create_index":"false",
"transport.tcp.compress":"true","discovery.zen.ping.multicast.enabled":
"false","name":".42:1"}},"g302MXTgRY2qG7OfTXbevQ":{"name":
".43:1","transport_address":"inet[/.43:9300]","hostname":
"bdm02-test","version":"0.90.3","http_address":"inet[/.43:9200]",
"settings":{"path.home":"/home/nli/opt/bdm","pidfile":
"/home/nli/opt/bdm/pid/.43:1/elasticsearch.pid","path.logs":
"/home/nli/opt/bdm/log/elasticsearch/.43:1","path.work":
"/tmp/elasticsearch/.43:1","max-open-files":"true","path.data":
"/home/nli/data/elasticsearchdata","config":
"/home/nli/opt/bdm/config/.43:1.yml",
"discovery.zen.ping.unicast.hosts.2":".44:9300",
"discovery.zen.ping.unicast.hosts.1":".43:9300",
"gateway.recovery_after_time":"5m","discovery.zen.ping.unicast.hosts.0":
".42:9300","indices.ttl.interval":"86400s","bootstrap.mlockall":"true",
"cluster.name":"","http.port":"9200","network.publish_host":".43",
"indices.memory.index_buffer_size":"20%",
"discovery.zen.minimum_master_nodes":"2","discovery.zen.fd.ping_timeout":
"60s","gateway.recovery_after_nodes":"3","gateway.expected_nodes":"3",
"discovery.zen.ping.timeout":"30s","indices.fielddata.cache.expire":"1m",
"discovery.zen.fd.ping_retries":"10","node.name":".43:1",
"indices.fielddata.cache.size":"10%","action.auto_create_index":"false",
"transport.tcp.compress":"true","discovery.zen.ping.multicast.enabled":
"false","name":".43:1"}},"WxvcZ71jSom2aklRu2LYjg":{"name":
".44:1","transport_address":"inet[/.44:9300]","hostname":
"bdm03-test","version":"0.90.3","http_address":"inet[/.44:9200]",
"settings":{"path.home":"/home/nli/opt/bdm","pidfile":
"/home/nli/opt/bdm/pid/.44:1/elasticsearch.pid","path.logs":
"/home/nli/opt/bdm/log/elasticsearch/.44:1","path.work":
"/tmp/elasticsearch/.44:1","max-open-files":"true","path.data":
"/home/nli/data/elasticsearchdata","config":
"/home/nli/opt/bdm/config/.44:1.yml",
"discovery.zen.ping.unicast.hosts.2":".44:9300",
"discovery.zen.ping.unicast.hosts.1":".43:9300",
"gateway.recovery_after_time":"5m","discovery.zen.ping.unicast.hosts.0":
".42:9300","indices.ttl.interval":"86400s","bootstrap.mlockall":"true",
"cluster.name":"","http.port":"9200","network.publish_host":".44",
"indices.memory.index_buffer_size":"20%",
"discovery.zen.minimum_master_nodes":"2","discovery.zen.fd.ping_timeout":
"60s","gateway.recovery_after_nodes":"3","gateway.expected_nodes":"3",
"discovery.zen.ping.timeout":"30s","indices.fielddata.cache.expire":"1m",
"discovery.zen.fd.ping_retries":"10","node.name":".44:1",
"indices.fielddata.cache.size":"10%","action.auto_create_index":"false",
"transport.tcp.compress":"true","discovery.zen.ping.multicast.enabled":
"false","name":".44:1"Enter code here...
Best,
Vahid
On Wednesday, February 12, 2014 2:45:46 PM UTC+1, Alexander Reelsen wrote:
>
> Hey
>
> minor question/just a wild guess: Could this one node (1gb is the default
> heap size) be a client node, where you didnot configure the mlockall
> setting as it is started inside of another java application? If the
> mlockall setting was successful, I highly doubt, that any of these
> processes swap.
>
> All the nodes connected to the cluster should show in the nodes info and
> have the mlockall setting set to true.
>
>
> --Alex
>
>
> On Tue, Feb 11, 2014 at 4:51 PM, Vahid >wrote:
>
>> Hi Alex,
>>
Re: Elasticsearch swapping
Hey minor question/just a wild guess: Could this one node (1gb is the default heap size) be a client node, where you didnot configure the mlockall setting as it is started inside of another java application? If the mlockall setting was successful, I highly doubt, that any of these processes swap. All the nodes connected to the cluster should show in the nodes info and have the mlockall setting set to true. --Alex On Tue, Feb 11, 2014 at 4:51 PM, Vahid wrote: > Hi Alex, > thank you. > > I've run the command and also it shows that mlockall is set to true ! > > On Tuesday, February 11, 2014 2:33:49 PM UTC+1, Alexander Reelsen wrote: > >> Hey, >> >> with recent elasticsearch versions (including newer 0.90), you can see if >> bootstrap.mlockall setting is really applied in the nodes info. So make >> sure setting it, was really successful. >> >> curl -XGET 'http://localhost:9200/_nodes' and search for mlockall, which >> must be set to true. >> >> >> --Alex >> >> >> On Tue, Feb 11, 2014 at 10:08 AM, Vahid wrote: >> >>> Thank you Tony and Mark, >>> >>> atm I have no more information about the virtualization because it's our >>> customer systems, maybe later I can provide more information regarding >>> that. >>> Other processes are our java applications which use ES to index and >>> search data. >>> >>> From htop/top I can see that almost all the swap memory is used, and by >>> running a bash script I can see how much swap space is used by which >>> process and mosthly is used by ES. >>> >>> >>> On Monday, February 10, 2014 10:44:00 PM UTC+1, Mark Walkom wrote: >>> How are you seeing this, ie what monitoring are you using? Also, what is the "~3 gb other processes" exactly for? Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 11 February 2014 04:16, Vahid wrote: > Hi, > > ES is running on a ubuntu 64bit in VM environment with the following > memory configuration: > > 12 gb total memory > 5 gb elasticsearch > ~3 gb other processes > and 4 gb left for OS. > > ES cluster configured with 3 nodes. > In the elasticsearch.yml bootstrap.mlockall set to true. On one of the > nodes I can see that ES is using about 1gb of swap space. > Also no warn log printed on the ES log file and only one node has this > problem. > > Is there any settings missed there? > > Many thanks, > > Vahid > > -- > You received this message because you are subscribed to the Google > Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to elasticsearc...@googlegroups.com. > > To view this discussion on the web visit https://groups.google.com/d/ > msgid/elasticsearch/e9953624-3f35-4b86-a779-ac58dd0e30ba%40goo > glegroups.com. > For more options, visit https://groups.google.com/groups/opt_out. > -- >>> You received this message because you are subscribed to the Google >>> Groups "elasticsearch" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to elasticsearc...@googlegroups.com. >>> To view this discussion on the web visit https://groups.google.com/d/ >>> msgid/elasticsearch/791e40ad-c688-4de4-af78-c1f68c0d9569% >>> 40googlegroups.com. >>> >>> For more options, visit https://groups.google.com/groups/opt_out. >>> >> >> -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to elasticsearch+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/b284a9c3-bf09-42e3-8393-8683e9d1425e%40googlegroups.com > . > > For more options, visit https://groups.google.com/groups/opt_out. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGCwEM_re%3DYNodKfXXu%2BpT6px-k8MFTSzZiZCAkKMdXx09LjkQ%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Elasticsearch swapping
Hi Alex, thank you. I've run the command and also it shows that mlockall is set to true ! On Tuesday, February 11, 2014 2:33:49 PM UTC+1, Alexander Reelsen wrote: > > Hey, > > with recent elasticsearch versions (including newer 0.90), you can see if > bootstrap.mlockall setting is really applied in the nodes info. So make > sure setting it, was really successful. > > curl -XGET 'http://localhost:9200/_nodes' and search for mlockall, which > must be set to true. > > > --Alex > > > On Tue, Feb 11, 2014 at 10:08 AM, Vahid >wrote: > >> Thank you Tony and Mark, >> >> atm I have no more information about the virtualization because it's our >> customer systems, maybe later I can provide more information regarding >> that. >> Other processes are our java applications which use ES to index and >> search data. >> >> From htop/top I can see that almost all the swap memory is used, and by >> running a bash script I can see how much swap space is used by which >> process and mosthly is used by ES. >> >> >> On Monday, February 10, 2014 10:44:00 PM UTC+1, Mark Walkom wrote: >> >>> How are you seeing this, ie what monitoring are you using? >>> Also, what is the "~3 gb other processes" exactly for? >>> >>> Regards, >>> Mark Walkom >>> >>> Infrastructure Engineer >>> Campaign Monitor >>> email: ma...@campaignmonitor.com >>> web: www.campaignmonitor.com >>> >>> >>> On 11 February 2014 04:16, Vahid wrote: >>> Hi, ES is running on a ubuntu 64bit in VM environment with the following memory configuration: 12 gb total memory 5 gb elasticsearch ~3 gb other processes and 4 gb left for OS. ES cluster configured with 3 nodes. In the elasticsearch.yml bootstrap.mlockall set to true. On one of the nodes I can see that ES is using about 1gb of swap space. Also no warn log printed on the ES log file and only one node has this problem. Is there any settings missed there? Many thanks, Vahid -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/ msgid/elasticsearch/e9953624-3f35-4b86-a779-ac58dd0e30ba% 40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. >>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to elasticsearc...@googlegroups.com . >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/791e40ad-c688-4de4-af78-c1f68c0d9569%40googlegroups.com >> . >> >> For more options, visit https://groups.google.com/groups/opt_out. >> > > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b284a9c3-bf09-42e3-8393-8683e9d1425e%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Elasticsearch swapping
Hey, with recent elasticsearch versions (including newer 0.90), you can see if bootstrap.mlockall setting is really applied in the nodes info. So make sure setting it, was really successful. curl -XGET 'http://localhost:9200/_nodes' and search for mlockall, which must be set to true. --Alex On Tue, Feb 11, 2014 at 10:08 AM, Vahid wrote: > Thank you Tony and Mark, > > atm I have no more information about the virtualization because it's our > customer systems, maybe later I can provide more information regarding > that. > Other processes are our java applications which use ES to index and search > data. > > From htop/top I can see that almost all the swap memory is used, and by > running a bash script I can see how much swap space is used by which > process and mosthly is used by ES. > > > On Monday, February 10, 2014 10:44:00 PM UTC+1, Mark Walkom wrote: > >> How are you seeing this, ie what monitoring are you using? >> Also, what is the "~3 gb other processes" exactly for? >> >> Regards, >> Mark Walkom >> >> Infrastructure Engineer >> Campaign Monitor >> email: ma...@campaignmonitor.com >> web: www.campaignmonitor.com >> >> >> On 11 February 2014 04:16, Vahid wrote: >> >>> Hi, >>> >>> ES is running on a ubuntu 64bit in VM environment with the following >>> memory configuration: >>> >>> 12 gb total memory >>> 5 gb elasticsearch >>> ~3 gb other processes >>> and 4 gb left for OS. >>> >>> ES cluster configured with 3 nodes. >>> In the elasticsearch.yml bootstrap.mlockall set to true. On one of the >>> nodes I can see that ES is using about 1gb of swap space. >>> Also no warn log printed on the ES log file and only one node has this >>> problem. >>> >>> Is there any settings missed there? >>> >>> Many thanks, >>> >>> Vahid >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "elasticsearch" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to elasticsearc...@googlegroups.com. >>> >>> To view this discussion on the web visit https://groups.google.com/d/ >>> msgid/elasticsearch/e9953624-3f35-4b86-a779-ac58dd0e30ba% >>> 40googlegroups.com. >>> For more options, visit https://groups.google.com/groups/opt_out. >>> >> >> -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to elasticsearch+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/791e40ad-c688-4de4-af78-c1f68c0d9569%40googlegroups.com > . > > For more options, visit https://groups.google.com/groups/opt_out. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGCwEM_qcrBLS3vw-GPTd1UuwyVKn3kN2ozpp14jCrE-iPZkwA%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Elasticsearch swapping
Thank you Tony and Mark, atm I have no more information about the virtualization because it's our customer systems, maybe later I can provide more information regarding that. Other processes are our java applications which use ES to index and search data. >From htop/top I can see that almost all the swap memory is used, and by running a bash script I can see how much swap space is used by which process and mosthly is used by ES. On Monday, February 10, 2014 10:44:00 PM UTC+1, Mark Walkom wrote: > > How are you seeing this, ie what monitoring are you using? > Also, what is the "~3 gb other processes" exactly for? > > Regards, > Mark Walkom > > Infrastructure Engineer > Campaign Monitor > email: ma...@campaignmonitor.com > web: www.campaignmonitor.com > > > On 11 February 2014 04:16, Vahid > wrote: > >> Hi, >> >> ES is running on a ubuntu 64bit in VM environment with the following >> memory configuration: >> >> 12 gb total memory >> 5 gb elasticsearch >> ~3 gb other processes >> and 4 gb left for OS. >> >> ES cluster configured with 3 nodes. >> In the elasticsearch.yml bootstrap.mlockall set to true. On one of the >> nodes I can see that ES is using about 1gb of swap space. >> Also no warn log printed on the ES log file and only one node has this >> problem. >> >> Is there any settings missed there? >> >> Many thanks, >> >> Vahid >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to elasticsearc...@googlegroups.com . >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/e9953624-3f35-4b86-a779-ac58dd0e30ba%40googlegroups.com >> . >> For more options, visit https://groups.google.com/groups/opt_out. >> > > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/791e40ad-c688-4de4-af78-c1f68c0d9569%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Elasticsearch swapping
How are you seeing this, ie what monitoring are you using? Also, what is the "~3 gb other processes" exactly for? Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: ma...@campaignmonitor.com web: www.campaignmonitor.com On 11 February 2014 04:16, Vahid wrote: > Hi, > > ES is running on a ubuntu 64bit in VM environment with the following > memory configuration: > > 12 gb total memory > 5 gb elasticsearch > ~3 gb other processes > and 4 gb left for OS. > > ES cluster configured with 3 nodes. > In the elasticsearch.yml bootstrap.mlockall set to true. On one of the > nodes I can see that ES is using about 1gb of swap space. > Also no warn log printed on the ES log file and only one node has this > problem. > > Is there any settings missed there? > > Many thanks, > > Vahid > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to elasticsearch+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/e9953624-3f35-4b86-a779-ac58dd0e30ba%40googlegroups.com > . > For more options, visit https://groups.google.com/groups/opt_out. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624ZGA%2BWxafxU8avh90EG%3DwP%2BOgOEhL%2BZ_tYLYd7icyGZAg%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Elasticsearch swapping
Although others might be able to comment, IMO you need to provide more information, eg Virtualization technology What else is running in each node. So, for comparison, this is my "lab" setup. I've been testing with approx. 1GB data that has created up to 15GB of additional metadata. I've been surprised how efficient and functional the cluster is even with its minimal resources for the amount of data it's working with almost no noticeable latencies (the current bottleneck is Host disk I/O, nothing in the Guests). I have observed in any of the Guests. es-hq states thus, and I have verified by running "free" in each node. node1 - 4GB ram, 20gb HDD, LXDE Desktop installed, so can run multiple consoles, web browser and more easily. Running ES, Kibana (and other web apps like es-head, es-hq), redis, logstash, netcat nodes 2-4 - 1GB RAM 20GB HDD All cloned from one so are as consistently the same as possible. All running in text mode (no Desktop) so that all stats and observations should be as unpolluted as possible Running ES I'm currently running in VMware, but IMO the configuration would be fine for practically any other virtualization technology. Note that if you're running a Desktop in every node, the OS may be pushing parts of the Desktop's memory usage to swap. Tony On Monday, February 10, 2014 9:16:58 AM UTC-8, Vahid wrote: > Hi, > > ES is running on a ubuntu 64bit in VM environment with the following > memory configuration: > > 12 gb total memory > 5 gb elasticsearch > ~3 gb other processes > and 4 gb left for OS. > > ES cluster configured with 3 nodes. > In the elasticsearch.yml bootstrap.mlockall set to true. On one of the > nodes I can see that ES is using about 1gb of swap space. > Also no warn log printed on the ES log file and only one node has this > problem. > > Is there any settings missed there? > > Many thanks, > > Vahid > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c33b766f-c3c5-4bda-8820-bf3eb57d625a%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
