Re: Multy-tenancy - level of service garantee
Regarding shards RAM allocation : - Since each shard comes with a Lucene instance : - If I have 2 shards, each belonging to a different index, each index being used by a different application. Given that each shard's Lucene highly uses OS cache, how can I certify that each Lucene will have enough OS cache for its magic to perform? Thanks, Le jeudi 12 février 2015 22:44:29 UTC+1, Mark Walkom a écrit : How do you guarantee a level of service provided any other way? Redundancy and smart planning and design. It's no different with ES. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5ca10ab5-8e25-4072-828f-7a4fafb3afdf%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Moving to production considerations - applications with their clusters' infrastructure
Hi Charlie, That's excellent news! Thank you for your slideshare and related github! Regards, Victor Le vendredi 13 février 2015 16:07:11 UTC+1, Charlie Hull a écrit : Hi, Firstly, thanks for the kind words about the performance study: we're hoping to revisit this soon after the feedback we've had on better tuning for each engine. I agree there's a paucity of studies, but in a month or two we should have one from a project we're working on to build an index of product data - we hope to present this at the London Meetup group but will publish the slides afterwards. Sorry this is in the future! Cheers Charlie On 12 February 2015 at 14:27, rondel...@gmail.com javascript: wrote: Hi everyone, I am considering moving one or several elasticsearch clusters to production. Although Elasticsearch's documentation and community is *great*, I am strongly startled not to find any *complete use-case story* stretching from application(s) needs and data considerations to hardware ones. Indeed, I understand why what/how much hardware / configuration / sharding questions are systematically replied with both it depends followed by test. But then, what about a few complete descriptions, out of so many elasticsearch users, from data use case to cluster's internals, along with a few performance and nodes stats? So here are questions, before moving to production : Are there any *complete* use cases around? Could you share some? By complete I mean including *at least some* of the following : 1. *Application needs and scope* 2. *Indexing Data indications* : data volume, documents mapping, documents / indexes volume 3. *Searching Data indications* : different applications, queries, use of facets - filters - aggregations, concurrent indexing 4. *Cluster Hardware* : machines' hardware (RAM, Disks/SSD - DAS-JBOD/SAN/NAS), JVM heap / OS Cache, nb of machines, back office network 5. *Cluster Configuration* : one or several indexes, sharding, replication, master nodes, data nodes, use of over-sharding at start-up, use of re-indexing 6. *Benchmaks *: queries response times, QPS, with or without concurrent indexing, memory heap sweet spot, nodes stats For those interested, here are the (not *complete*) best-among-very-few exemples I've stumbled upon so far : - The very best (perfs with hardware and query description) : http://fr.slideshare.net/charliejuggler/lucene-solrlondonug-meetup28nov2014-solr-es-performance - Hardware and master nodes heap : https://groups.google.com/forum/?fromgroups#!searchin/elasticsearch/sizing/elasticsearch/V5BtrCGOqoU/l7x6vqMEx5YJ - *6th slide* - Hardware and storage with number of documents (well, without indexes and documents storage volume nor RAM consumption) : https://speakerdeck.com/bhaskarvk/scaling-elasticsearch-washington-dc-meetup With JBOD / SAN storage discussion in To Raid or not to Raid: https://groups.google.com/forum/?fromgroups#!searchin/elasticsearch/hardware/elasticsearch/HSj2fZGdU1Y/4mFCBTCb-JcJ - Usual heap considerations in a real case : https://codeascraft.com/2014/12/04/juggling-multiple-elasticsearch-instances-on-a-single-host/ Do not forget Elasticsearch awesome docs for moving to production considerations : - http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/administration.html - http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/deploy.html - *http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/hardware.html http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/hardware.html* - *http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/heap-sizing.html http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/heap-sizing.html* -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com javascript:. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/927f60b1-8ae2-463e-b725-5f4f993905d9%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/927f60b1-8ae2-463e-b725-5f4f993905d9%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/34acb9f4-a622-4ed6-b877-d605f0c6382b%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Moving to production considerations - applications with their clusters' infrastructure
Hi everyone, I am considering moving one or several elasticsearch clusters to production. Although Elasticsearch's documentation and community is *great*, I am strongly startled not to find any *complete use-case story* stretching from application(s) needs and data considerations to hardware ones. Indeed, I understand why what/how much hardware / configuration / sharding questions are systematically replied with both it depends followed by test. But then, what about a few complete descriptions, out of so many elasticsearch users, from data use case to cluster's internals, along with a few performance and nodes stats? So here are questions, before moving to production : Are there any *complete* use cases around? Could you share some? By complete I mean including *at least some* of the following : 1. *Application needs and scope* 2. *Indexing Data indications* : data volume, documents mapping, documents / indexes volume 3. *Searching Data indications* : different applications, queries, use of facets - filters - aggregations, concurrent indexing 4. *Cluster Hardware* : machines' hardware (RAM, Disks/SSD - DAS-JBOD/SAN/NAS), JVM heap / OS Cache, nb of machines, back office network 5. *Cluster Configuration* : one or several indexes, sharding, replication, master nodes, data nodes, use of over-sharding at start-up, use of re-indexing 6. *Benchmaks *: queries response times, QPS, with or without concurrent indexing, memory heap sweet spot, nodes stats For those interested, here are the (not *complete*) best-among-very-few exemples I've stumbled upon so far : - The very best (perfs with hardware and query description) : http://fr.slideshare.net/charliejuggler/lucene-solrlondonug-meetup28nov2014-solr-es-performance - Hardware and master nodes heap : https://groups.google.com/forum/?fromgroups#!searchin/elasticsearch/sizing/elasticsearch/V5BtrCGOqoU/l7x6vqMEx5YJ - *6th slide* - Hardware and storage with number of documents (well, without indexes and documents storage volume nor RAM consumption) : https://speakerdeck.com/bhaskarvk/scaling-elasticsearch-washington-dc-meetup With JBOD / SAN storage discussion in To Raid or not to Raid: https://groups.google.com/forum/?fromgroups#!searchin/elasticsearch/hardware/elasticsearch/HSj2fZGdU1Y/4mFCBTCb-JcJ - Usual heap considerations in a real case : https://codeascraft.com/2014/12/04/juggling-multiple-elasticsearch-instances-on-a-single-host/ Do not forget Elasticsearch awesome docs for moving to production considerations : - http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/administration.html - http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/deploy.html - *http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/hardware.html* - *http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/heap-sizing.html* -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/927f60b1-8ae2-463e-b725-5f4f993905d9%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Use cases - Production examples: datas, queries, cluster hardware and conf, and statistics
Hi everyone, I am considering moving one or several elasticsearch clusters to production. Although Elasticsearch's documentation and community is *great*, I am strongly startled not to find any *complete use-case story* stretching from application(s) needs and data considerations to hardware ones. Indeed, I understand why what/how much hardware / configuration / sharding questions are systematically replied with both it depends followed by test. But then, what about a few complete descriptions, out of so many elasticsearch users, from data use case to cluster's internals, along with a few performance and nodes stats? So here are questions, before moving to production : Are there any *complete* use cases around? Could you share some? By complete I mean including *at least some* of the following : 1. *Application needs and scope* 2. *Indexing Data indications* : data volume, documents mapping, documents / indexes volume 3. *Searching Data indications* : different applications, queries, use of facets - filters - aggregations, concurrent indexing 4. *Cluster Hardware* : machines' hardware (RAM, Disks/SSD - DAS-JBOD/SAN/NAS), JVM heap / OS Cache, nb of machines, back office network 5. *Cluster Configuration* : one or several indexes, sharding, replication, master nodes, data nodes, use of over-sharding at start-up, use of re-indexing 6. *Benchmaks *: queries response times, QPS, with or without concurrent indexing, memory heap sweet spot, nodes stats For those interested, here are the (not *complete*) best-among-very-few exemples I've stumbled upon so far : - The very best (perfs with hardware and query description) : http://fr.slideshare.net/charliejuggler/lucene-solrlondonug-meetup28nov2014-solr-es-performance - Hardware and master nodes heap : https://groups.google.com/forum/?fromgroups#!searchin/elasticsearch/sizing/elasticsearch/V5BtrCGOqoU/l7x6vqMEx5YJ - *6th slide* - Hardware and storage with number of documents (well, without indexes and documents storage volume nor RAM consumption) : https://speakerdeck.com/bhaskarvk/scaling-elasticsearch-washington-dc-meetup With JBOD / SAN storage discussion in To Raid or not to Raid: https://groups.google.com/forum/?fromgroups#!searchin/elasticsearch/hardware/elasticsearch/HSj2fZGdU1Y/4mFCBTCb-JcJ - Usual heap considerations in a real case : https://codeascraft.com/2014/12/04/juggling-multiple-elasticsearch-instances-on-a-single-host/ Do not forget Elasticsearch awesome docs for moving to production considerations : - http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/administration.html - http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/deploy.html - *http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/hardware.html http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/hardware.html* - *http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/heap-sizing.html http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/heap-sizing.html* -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/57c967ea-8bf0-4dce-a7ca-4a746ee21250%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Multy-tenancy - level of service garantee
Hi everyone, After my precedent question https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/US-BA4R_Qdc regarding examples of clusters in production, I am wondering about multy-tenancy and garantee of service in Elasticsearch : *Multy-tenant cluster* : Is there a way to *garantee a level of service* / capacity planning for *each tenant* using the cluster (its *own indexes*) ? Thanks, -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7f99a562-ae18-447f-b227-cd145483dcf3%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Moving to production - Storage and servers
Hi everyone, After my first https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/US-BA4R_Qdc and second question https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/FGKUmzn-WSs regarding clusters' examples and multy-tenancy with garantee of service, I have one more on my way to production : Could you describe the *pros* and *cons* of the : 1. Disks : *DAS* (Direct Attached Storage) - *JBOD* (Just a Bunch Of Disks) / *SAN* (Storage Area Network)? I take it that NAS is most of the time to be avoided 2. Servers : bare metal servers / virtual servers ? For those interested : - JBOD / SAN storage discussion in To Raid or not to Raid: https://groups.google.com/forum/?fromgroups#!searchin/elasticsearch/hardware/elasticsearch/HSj2fZGdU1Y/4mFCBTCb-JcJ - Doc which states about SSDs, RAID0, NAS, networks http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/hardware.html Thanks! -- You received this message because you are subscribed to the Google Groups elasticsearch group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3ab36482-e4fb-4dd0-99cd-044d2e4257ea%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
