Tim, For the storage backing Elasticsearch data nodes, we have been using one 500GB Cinder volume (backed by a Ceph cluster built from DL380s filled with 1.2TB 10k SAS drives) per data node. However, we've found that a VM with 8 vCPU and 64GB RAM can make use of more than 500GB disk capacity without bottlenecking on CPU or memory, so we are experimenting with 1TB or 1.5TB options per data node. We are also moving to a different tier of storage that uses an array of SSDs instead of spinning rust, though this change has very little to do with performance and very much to do with the automatic deduplication, compression, and encryption offered by the hardware backend (Solidfire) for that storage tier. # <-- Not a vendor promo; just letting you know what we're using for that tier.
We get a lot of duplicated messages in Elasticsearch since we're using if for log monitoring, and JSON documents compress very well... so it actually costs us significantly less to leverage a storage hardware platform that provides native deduplication and compression. Having SSDs in the mix probably helps reduce latency a bit (due to lower seek times), but honestly we didn't have enough of a latency problem to justify moving away from volumes backed by Ceph. Guidance from Elastic is always going to advocate using local SSDs when possible, but I'm pretty sure that's not what Elastic uses for their own cloud offering... Thanks, Nathan Randall From: Tim Bell <tim.b...@cern.ch<mailto:tim.b...@cern.ch>> Date: Saturday, September 3, 2016 at 1:12 AM To: David Medberry <openst...@medberry.net<mailto:openst...@medberry.net>> Cc: openstack-operators <openstack-operators@lists.openstack.org<mailto:openstack-operators@lists.openstack.org>> Subject: Re: [Openstack-operators] ElasticSearch on OpenStack Thanks. How's the storage handled ? We're seeing slow I/O on local storage (which is also limited on space) and latencies with Ceph for block storage. Tim From: <medbe...@gmail.com<mailto:medbe...@gmail.com>> on behalf of David Medberry <openst...@medberry.net<mailto:openst...@medberry.net>> Date: Friday 2 September 2016 at 22:18 To: Tim Bell <tim.b...@cern.ch<mailto:tim.b...@cern.ch>> Cc: openstack-operators <openstack-operators@lists.openstack.org<mailto:openstack-operators@lists.openstack.org>> Subject: Re: [Openstack-operators] ElasticSearch on OpenStack Nathan: The page at https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html gives you good advice on a maximum size for the elasticsearch VM's memory. Nathan: suggest you pick a flavor with 64GB RAM or less, then base other sizing things off of that (i.e. choose a flavor with 64GB of RAM and as many CPUs as possible for that RAM allocation, then base disk size on testing of your use case) Nathan: give java heap 30GB, and leave the rest of the memory to the OS filesystem cache so that Lucene can make best use of it. Nathan: that's mostly it for tuning. elasticsearch publishes many other docs for tuning recommendations, but there isn't anything specific to openstack besides the flavor choice. i personally chose CPU size (8CPU) such that all vCPUs for each VM would fit on a single NUMA node, which is a best practice for ESXi but not sure if it applies to KVM. (resending for clarity) On Fri, Sep 2, 2016 at 6:46 AM, David Medberry <openst...@medberry.net<mailto:openst...@medberry.net>> wrote: Hey Tim, We've just started this effort. I'll see if the guy running the service can comment today. On Fri, Sep 2, 2016 at 6:36 AM, Tim Bell <tim.b...@cern.ch<mailto:tim.b...@cern.ch>> wrote: Has anyone had experience running ElasticSearch on top of OpenStack VMs ? Are there any tuning recommendations ? Thanks Tim _______________________________________________ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org<mailto:OpenStack-operators@lists.openstack.org> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
_______________________________________________ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
