Sorry, correction, we were running ES and LS with java version "1.7.0_51", I just looked up the default on that system, which (someone says) is required for some other applications... but yes, we were using the correct Java version! :)
-Robin- On 25 March 2014 08:41, Mark Walkom <[email protected]> wrote: > Ouch, why such an old version of java? (And is is Open or Oracle?) > v7 is the minimum for ES and LS. > > Regards, > Mark Walkom > > Infrastructure Engineer > Campaign Monitor > email: [email protected] > web: www.campaignmonitor.com > > > On 25 March 2014 18:20, Robin Clarke <[email protected]> wrote: > >> Thanks for the tip with the number of masters! >> >> java version "1.6.0_45" on Debian 3.2.54-2 >> >> >> On 25 March 2014 07:55, Mark Walkom <[email protected]> wrote: >> >>> Java version? Also what OS? >>> >>> Just as a general note, it's always good to have an uneven number of >>> masters to ensure you get a majority quorum. >>> >>> Regards, >>> Mark Walkom >>> >>> Infrastructure Engineer >>> Campaign Monitor >>> email: [email protected] >>> web: www.campaignmonitor.com >>> >>> >>> On 25 March 2014 17:35, Robin Clarke <[email protected]> wrote: >>> >>>> Each node had 8 cores (2.4GHz Xeon), 32GB RAM, SSD disks (I never saw >>>> IOWait, but was also focusing on ingestion rate). >>>> I always had 2 master nodes, and in addition tried the configurations >>>> 20, 10 and 5 data nodes. >>>> Running Elasticsearch 1.0.1 (but with Logstash 1.3.3) >>>> >>>> -Robin- >>>> >>>> >>>> On 25 March 2014 06:17, Mark Walkom <[email protected]> wrote: >>>> >>>>> Can you elaborate what your cluster setup is like? >>>>> >>>>> Node specs (disk, RAM, CPU), how many master/data nodes and what >>>>> version of ES and java you're running? >>>>> >>>>> Regards, >>>>> Mark Walkom >>>>> >>>>> Infrastructure Engineer >>>>> Campaign Monitor >>>>> email: [email protected] >>>>> web: www.campaignmonitor.com >>>>> >>>>> >>>>> On 25 March 2014 16:11, Robin Clarke <[email protected]> wrote: >>>>> >>>>>> I did some intensive tests last week on a 20-node cluster and had >>>>>> the following insights - I'd be interested if anyone has >>>>>> similar/dissimilar >>>>>> experience. >>>>>> The had 20 nodes had 8 cores each, and 32GB memory each. I set up >>>>>> Elasticsearch to have 15GB of that memory. >>>>>> The sample events I was using were Apache logs (common format) >>>>>> without any additional fields (no geoip, useragent etc. plugins). >>>>>> When running as a 20-node cluster, I got a maximum igestion rate of >>>>>> 2500k events/minute (41k/second), *but* the bottleneck was the >>>>>> logstash CPU load... so I reduced to a 10 node cluster... >>>>>> With the 10 nodes I initially had 1600k/minute (27k) and acheived >>>>>> 1800k/minute (30k/second) by increasing index_refresh_interval to 30s and >>>>>> index_buffer_size to 20% >>>>>> Further reducing to 5 nodes, I had 1100k/minute (18k/second). >>>>>> This brings me to an interesting comparison: at 10 nodes, I have 3k >>>>>> events/second/node, and with 5 nodes I have 3.66k events/second/node. >>>>>> i.e. >>>>>> the overhead for doubling the number of nodes from 5 to 10 is about 20%. >>>>>> Is this to be expected? Just how scalable is Elasticsearch - at what >>>>>> point is the diminishing return on adding nodes not cost effective? >>>>>> Is the further logical reduction to 375 events/core/second still >>>>>> meaningful? >>>>>> >>>>>> Cheers, >>>>>> -Robin- >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "elasticsearch" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/elasticsearch/3353dab5-6241-4b41-8845-6c5f8553d488%40googlegroups.com<https://groups.google.com/d/msgid/elasticsearch/3353dab5-6241-4b41-8845-6c5f8553d488%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to a topic in the >>>>> Google Groups "elasticsearch" group. >>>>> To unsubscribe from this topic, visit >>>>> https://groups.google.com/d/topic/elasticsearch/muIKhFkrxFc/unsubscribe >>>>> . >>>>> To unsubscribe from this group and all its topics, send an email to >>>>> [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/elasticsearch/CAEM624b8a8ZZbOc81g18XZWNz3qYV3gby%3DCtGrPZ8f%2BwiXXYEg%40mail.gmail.com<https://groups.google.com/d/msgid/elasticsearch/CAEM624b8a8ZZbOc81g18XZWNz3qYV3gby%3DCtGrPZ8f%2BwiXXYEg%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> >>>> >>>> -- >>>> Best winds, >>>> -Robin- >>>> ~:) >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "elasticsearch" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/elasticsearch/CACX78vaf_r-GY6KVMk95R-jip-aMdr2y20-WwNL84jQe0PvD1Q%40mail.gmail.com<https://groups.google.com/d/msgid/elasticsearch/CACX78vaf_r-GY6KVMk95R-jip-aMdr2y20-WwNL84jQe0PvD1Q%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "elasticsearch" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/elasticsearch/muIKhFkrxFc/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elasticsearch/CAEM624audypF5E%2BDyZcUJmf%2BtddunbLswKPwEJMyHQ_X%2B2xdEw%40mail.gmail.com<https://groups.google.com/d/msgid/elasticsearch/CAEM624audypF5E%2BDyZcUJmf%2BtddunbLswKPwEJMyHQ_X%2B2xdEw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> >> -- >> Best winds, >> -Robin- >> ~:) >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/CACX78vYbV7%2BJ7c7_aSjzh7UL4e3vykgws9pFMevjZetarcEGJA%40mail.gmail.com<https://groups.google.com/d/msgid/elasticsearch/CACX78vYbV7%2BJ7c7_aSjzh7UL4e3vykgws9pFMevjZetarcEGJA%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/elasticsearch/muIKhFkrxFc/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/CAEM624Zv19Ww7qYthS9ti1XgNF3E0e2eFQ8dsf0aPLA9Z-Mgrw%40mail.gmail.com<https://groups.google.com/d/msgid/elasticsearch/CAEM624Zv19Ww7qYthS9ti1XgNF3E0e2eFQ8dsf0aPLA9Z-Mgrw%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- Best winds, -Robin- ~:) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CACX78vbgqVXOxpNDTVYN4EF_Yg9C_04MZUg8uNxrewmA%2BBbo5Q%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
