subject:"Performance Limitation with ELK stack"

Re: Performance Limitation with ELK stack

2015-02-09 Thread Hagai T

Hi Itamar,

In my case it doesn't seems to be a CPU bottleneck at all.
The instance is actually doing nothing and this is what I don't understand.
I tried to tweak some of the configurations like workers but more than 4 
doesn't seem to improve anything.

Maybe I am missing something?

On Monday, February 9, 2015 at 12:22:28 PM UTC+2, Itamar Syn-Hershko wrote:
>
> Logstash is CPU bound, SSD won't help. It's a JRuby implementation. Try to 
> see if you can have multiple logstash shippers on the same logs. Having a 
> redis / kafka server as a middle tier is also a general practice. If that 
> is not feasible then yes - my advise to you would be to roll your own.
>
> --
>
> Itamar Syn-Hershko
> http://code972.com | @synhershko 
> Freelance Developer & Consultant
> Lucene.NET committer and PMC member
>
> On Mon, Feb 9, 2015 at 12:15 PM, Hagai T > 
> wrote:
>
>> Hi,
>>
>> We were able to identify the bottleneck which seems to be the Logstash 
>> service.
>> It seems that the Elasticsearch cluster is able to handle 40,000 per 
>> second documents with a 3 ES servers cluster using Java client that was 
>> written by us using SDK with bulk inserts.
>> The client (written for load testing) is generating JSON format and send 
>> it to Elasticsearch for further processing.
>>
>> We run the same test with Logstash which reads the JSON format from 
>> Apache access log on a general purpose SSD and managed to achieve maximum 
>> of 4,000 requests per second.
>> With 2 Logstash servers we achieved 8,000 req per second.
>>
>> Getting rid of the filtering section in logstash configuration file 
>> helped us get to this number. with filtering we achieved only 1,5000-2,000 
>> req per sec.
>> I also tried to move the log file to Ephemeral storage but didn't get any 
>> improvement.
>> We don't have any resources problem in the Logstash server (I/O / CPU) so 
>> it seems like a limit in the file input module or either logstash itself.
>>
>> I was able to test Logstash performance by creating huge log file 2GB and 
>> starting Logstash to send it's content. 
>> I also did tried with smaller files (4-5MB each) but performance didn't 
>> get any better.
>>
>> Does it sound reasonable for you guys that I got to a limit of 4,000 req 
>> per second with one Logstash?
>> If you have any suggestions of how to proceed from here I will be more 
>> than happy to hear that.
>>
>> If we can't get more from one Logstash, we'll have to develop our own 
>> Java service to do that instead.
>>
>> *Apache Access log file output example - (already in JSON Format):*
>> { "timestamp":"2015-02-09T10:07:48+", 
>> "bq_timestamp":"2015-02-09T10:07:48", "client_ip":"52.2.11.111", 
>> "client_port":"80", "latency_ms":"57", "latency_sec":"0", 
>> "elb_status_code":"200", 
>> "request":"/il.html?e=fpAdOpportunity&w=wfl_dose&vid=1&vname=compName_PM&ecpm=8&adid=1814157&media_file_type=MEDIA_FILE_TYPE&media_file_url=MEDIA_FILE_URL¤t_url=%0Ahttp%3A%2F%
>> 2Fu-sd.gga.tv
>> %2Fa%2Fh%2FJvf82UX3%2Beff48Z%2fwU20swbapQoWau_%3Fcb%3D5605126933660359000%26pet%3Dpreroll%26pageUrl%3Dhttp%253A%252F%252F3ffese.com%26eov%3Deov%0A%09¤t_main_vast_url=MAIN_VAST_URL&error_code=ERROR_CODE&error_message=ERROR_MESSAGE&q9=
>> dsdase.com&apid=dose.com&d=Convert&device=6719&csize=300X250&token=14123669&cb=260174713417&pc=PLAYCOUNT",
>>  
>> "request_path":"/il.html", "referer":"-", "user_agent":"Mozilla/5.0 
>> (redhat-x86_64-linux-gnu) Siege/3.0.8" }
>>
>>
>> *Logstash configuration file (for the testing I ran it with root without 
>> any limitations):*
>> input {
>> file {
>> path => 
>> "/var/log/httpd/aaa.d.com.logstash-acc.log.[0-9]*"
>> codec => json
>> type => "tracking"
>> discover_interval => 1
>> sincedb_path => "/opt/logstash/httpd-sincedb"
>> sincedb_write_interval => 1
>>   }
>> }
>>
>> output {
>> elasticsearch {
>> workers => 1
>> host => "aaa..com"
>> index => "%{request_path}-logstash-%{+-MM-dd}"
>> flush_size => 1000
>> cluster => "video"
>> codec => json
>> }
>> }
>>
>>
>> Your help is appreciated!
>> Thanks!
>>
>>
>>
>> On Thursday, February 5, 2015 at 1:56:34 PM UTC+2, Itamar Syn-Hershko 
>> wrote:
>>>
>>> I'd recommend you use ephemeral SSD - 2+ factor replicas and proper use 
>>> of the snapshot/restore API will provide you HA and DR guarantees.
>>>
>>> The rejections you are seeing are due to slow I/O operations, because 
>>> the disk is not local. There is a way to have a bigger queue but I'd advise 
>>> against that and instead go with a local fast disk.
>>>
>>> --
>>>
>>> Itamar Syn-Hershko
>>> http://code972.com | @synhershko 
>>> Freelance Developer & Consultant
>>> Lucene.NET committer and PMC member
>>>
>>> On Thu, Feb 5, 2015 at 1:51 PM, Hagai T  wrote

Re: Performance Limitation with ELK stack

2015-02-09 Thread Itamar Syn-Hershko

Logstash is CPU bound, SSD won't help. It's a JRuby implementation. Try to
see if you can have multiple logstash shippers on the same logs. Having a
redis / kafka server as a middle tier is also a general practice. If that
is not feasible then yes - my advise to you would be to roll your own.

--

Itamar Syn-Hershko
http://code972.com | @synhershko 
Freelance Developer & Consultant
Lucene.NET committer and PMC member

On Mon, Feb 9, 2015 at 12:15 PM, Hagai T  wrote:

> Hi,
>
> We were able to identify the bottleneck which seems to be the Logstash
> service.
> It seems that the Elasticsearch cluster is able to handle 40,000 per
> second documents with a 3 ES servers cluster using Java client that was
> written by us using SDK with bulk inserts.
> The client (written for load testing) is generating JSON format and send
> it to Elasticsearch for further processing.
>
> We run the same test with Logstash which reads the JSON format from Apache
> access log on a general purpose SSD and managed to achieve maximum of 4,000
> requests per second.
> With 2 Logstash servers we achieved 8,000 req per second.
>
> Getting rid of the filtering section in logstash configuration file helped
> us get to this number. with filtering we achieved only 1,5000-2,000 req per
> sec.
> I also tried to move the log file to Ephemeral storage but didn't get any
> improvement.
> We don't have any resources problem in the Logstash server (I/O / CPU) so
> it seems like a limit in the file input module or either logstash itself.
>
> I was able to test Logstash performance by creating huge log file 2GB and
> starting Logstash to send it's content.
> I also did tried with smaller files (4-5MB each) but performance didn't
> get any better.
>
> Does it sound reasonable for you guys that I got to a limit of 4,000 req
> per second with one Logstash?
> If you have any suggestions of how to proceed from here I will be more
> than happy to hear that.
>
> If we can't get more from one Logstash, we'll have to develop our own Java
> service to do that instead.
>
> *Apache Access log file output example - (already in JSON Format):*
> { "timestamp":"2015-02-09T10:07:48+",
> "bq_timestamp":"2015-02-09T10:07:48", "client_ip":"52.2.11.111",
> "client_port":"80", "latency_ms":"57", "latency_sec":"0",
> "elb_status_code":"200",
> "request":"/il.html?e=fpAdOpportunity&w=wfl_dose&vid=1&vname=compName_PM&ecpm=8&adid=1814157&media_file_type=MEDIA_FILE_TYPE&media_file_url=MEDIA_FILE_URL¤t_url=%0Ahttp%3A%2F%
> 2Fu-sd.gga.tv
> %2Fa%2Fh%2FJvf82UX3%2Beff48Z%2fwU20swbapQoWau_%3Fcb%3D5605126933660359000%26pet%3Dpreroll%26pageUrl%3Dhttp%253A%252F%252F3ffese.com%26eov%3Deov%0A%09¤t_main_vast_url=MAIN_VAST_URL&error_code=ERROR_CODE&error_message=ERROR_MESSAGE&q9=
> dsdase.com&apid=dose.com&d=Convert&device=6719&csize=300X250&token=14123669&cb=260174713417&pc=PLAYCOUNT",
> "request_path":"/il.html", "referer":"-", "user_agent":"Mozilla/5.0
> (redhat-x86_64-linux-gnu) Siege/3.0.8" }
>
>
> *Logstash configuration file (for the testing I ran it with root without
> any limitations):*
> input {
> file {
> path =>
> "/var/log/httpd/aaa.d.com.logstash-acc.log.[0-9]*"
> codec => json
> type => "tracking"
> discover_interval => 1
> sincedb_path => "/opt/logstash/httpd-sincedb"
> sincedb_write_interval => 1
>   }
> }
>
> output {
> elasticsearch {
> workers => 1
> host => "aaa..com"
> index => "%{request_path}-logstash-%{+-MM-dd}"
> flush_size => 1000
> cluster => "video"
> codec => json
> }
> }
>
>
> Your help is appreciated!
> Thanks!
>
>
>
> On Thursday, February 5, 2015 at 1:56:34 PM UTC+2, Itamar Syn-Hershko
> wrote:
>>
>> I'd recommend you use ephemeral SSD - 2+ factor replicas and proper use
>> of the snapshot/restore API will provide you HA and DR guarantees.
>>
>> The rejections you are seeing are due to slow I/O operations, because the
>> disk is not local. There is a way to have a bigger queue but I'd advise
>> against that and instead go with a local fast disk.
>>
>> --
>>
>> Itamar Syn-Hershko
>> http://code972.com | @synhershko 
>> Freelance Developer & Consultant
>> Lucene.NET committer and PMC member
>>
>> On Thu, Feb 5, 2015 at 1:51 PM, Hagai T  wrote:
>>
>>> Hi Itamar, thank you for the reply.
>>>
>>> This is 15k inserts totally and not for one host in the cluster.
>>> Yes, we have 15 sharding whiting one index. shards are spreaded on the
>>> nodes equally (automatically by Elasticsearch cluster).
>>> We currently use general purpose SSD and not Ephemeral storage.
>>>
>>> In addition, I see a lot of thread pool bulk rejections from the
>>> Elasticsearch side.
>>>
>>>
>>> On Thursday, February 5, 2015 at 1:33:05 PM UTC+2, Itamar Syn-Hershko
>>> wrote:

 What

Re: Performance Limitation with ELK stack

2015-02-09 Thread Hagai T

Hi,

We were able to identify the bottleneck which seems to be the Logstash 
service.
It seems that the Elasticsearch cluster is able to handle 40,000 per second 
documents with a 3 ES servers cluster using Java client that was written by 
us using SDK with bulk inserts.
The client (written for load testing) is generating JSON format and send it 
to Elasticsearch for further processing.

We run the same test with Logstash which reads the JSON format from Apache 
access log on a general purpose SSD and managed to achieve maximum of 4,000 
requests per second.
With 2 Logstash servers we achieved 8,000 req per second.

Getting rid of the filtering section in logstash configuration file helped 
us get to this number. with filtering we achieved only 1,5000-2,000 req per 
sec.
I also tried to move the log file to Ephemeral storage but didn't get any 
improvement.
We don't have any resources problem in the Logstash server (I/O / CPU) so 
it seems like a limit in the file input module or either logstash itself.

I was able to test Logstash performance by creating huge log file 2GB and 
starting Logstash to send it's content. 
I also did tried with smaller files (4-5MB each) but performance didn't get 
any better.

Does it sound reasonable for you guys that I got to a limit of 4,000 req 
per second with one Logstash?
If you have any suggestions of how to proceed from here I will be more than 
happy to hear that.

If we can't get more from one Logstash, we'll have to develop our own Java 
service to do that instead.

*Apache Access log file output example - (already in JSON Format):*
{ "timestamp":"2015-02-09T10:07:48+", 
"bq_timestamp":"2015-02-09T10:07:48", "client_ip":"52.2.11.111", 
"client_port":"80", "latency_ms":"57", "latency_sec":"0", 
"elb_status_code":"200", 
"request":"/il.html?e=fpAdOpportunity&w=wfl_dose&vid=1&vname=compName_PM&ecpm=8&adid=1814157&media_file_type=MEDIA_FILE_TYPE&media_file_url=MEDIA_FILE_URL¤t_url=%0Ahttp%3A%2F%2Fu-sd.gga.tv%2Fa%2Fh%2FJvf82UX3%2Beff48Z%2fwU20swbapQoWau_%3Fcb%3D5605126933660359000%26pet%3Dpreroll%26pageUrl%3Dhttp%253A%252F%252F3ffese.com%26eov%3Deov%0A%09¤t_main_vast_url=MAIN_VAST_URL&error_code=ERROR_CODE&error_message=ERROR_MESSAGE&q9=dsdase.com&apid=dose.com&d=Convert&device=6719&csize=300X250&token=14123669&cb=260174713417&pc=PLAYCOUNT",

"request_path":"/il.html", "referer":"-", "user_agent":"Mozilla/5.0 
(redhat-x86_64-linux-gnu) Siege/3.0.8" }

*Logstash configuration file (for the testing I ran it with root without 
any limitations):*
input {
file {
path => 
"/var/log/httpd/aaa.d.com.logstash-acc.log.[0-9]*"
codec => json
type => "tracking"
discover_interval => 1
sincedb_path => "/opt/logstash/httpd-sincedb"
sincedb_write_interval => 1
  }
}

output {
elasticsearch {
workers => 1
host => "aaa..com"
index => "%{request_path}-logstash-%{+-MM-dd}"
flush_size => 1000
cluster => "video"
codec => json
}
}

Your help is appreciated!
Thanks!

On Thursday, February 5, 2015 at 1:56:34 PM UTC+2, Itamar Syn-Hershko wrote:
>
> I'd recommend you use ephemeral SSD - 2+ factor replicas and proper use of 
> the snapshot/restore API will provide you HA and DR guarantees.
>
> The rejections you are seeing are due to slow I/O operations, because the 
> disk is not local. There is a way to have a bigger queue but I'd advise 
> against that and instead go with a local fast disk.
>
> --
>
> Itamar Syn-Hershko
> http://code972.com | @synhershko 
> Freelance Developer & Consultant
> Lucene.NET committer and PMC member
>
> On Thu, Feb 5, 2015 at 1:51 PM, Hagai T > 
> wrote:
>
>> Hi Itamar, thank you for the reply.
>>
>> This is 15k inserts totally and not for one host in the cluster.
>> Yes, we have 15 sharding whiting one index. shards are spreaded on the 
>> nodes equally (automatically by Elasticsearch cluster).
>> We currently use general purpose SSD and not Ephemeral storage.
>>
>> In addition, I see a lot of thread pool bulk rejections from the 
>> Elasticsearch side.
>>
>>
>> On Thursday, February 5, 2015 at 1:33:05 PM UTC+2, Itamar Syn-Hershko 
>> wrote:
>>>
>>> What is the question?...
>>>
>>> 15k inserts per sec per node is actually quite nice.
>>>
>>> Are your index sharded? If you write to one index only, you write to 
>>> maximum of x nodes where x is the number of shards of that index. Since 
>>> shards of the same index can co-exist on one node, check if you are 
>>> spanning writes.
>>>
>>> Use local disks - never EBS, and if you really care about writing speeds 
>>> use SSDs.
>>>
>>> Other than that, Mike did an excellent write up on the subject : 
>>> http://www.elasticsearch.org/blog/performance-
>>> considerations-elasticsearch-indexing/
>>>
>>> --
>>>
>>> Itamar Syn-Hershko
>>> http://code972.co

Re: Performance Limitation with ELK stack

2015-02-05 Thread Itamar Syn-Hershko

I'd recommend you use ephemeral SSD - 2+ factor replicas and proper use of
the snapshot/restore API will provide you HA and DR guarantees.

The rejections you are seeing are due to slow I/O operations, because the
disk is not local. There is a way to have a bigger queue but I'd advise
against that and instead go with a local fast disk.

--

Itamar Syn-Hershko
http://code972.com | @synhershko 
Freelance Developer & Consultant
Lucene.NET committer and PMC member

On Thu, Feb 5, 2015 at 1:51 PM, Hagai T  wrote:

> Hi Itamar, thank you for the reply.
>
> This is 15k inserts totally and not for one host in the cluster.
> Yes, we have 15 sharding whiting one index. shards are spreaded on the
> nodes equally (automatically by Elasticsearch cluster).
> We currently use general purpose SSD and not Ephemeral storage.
>
> In addition, I see a lot of thread pool bulk rejections from the
> Elasticsearch side.
>
>
> On Thursday, February 5, 2015 at 1:33:05 PM UTC+2, Itamar Syn-Hershko
> wrote:
>>
>> What is the question?...
>>
>> 15k inserts per sec per node is actually quite nice.
>>
>> Are your index sharded? If you write to one index only, you write to
>> maximum of x nodes where x is the number of shards of that index. Since
>> shards of the same index can co-exist on one node, check if you are
>> spanning writes.
>>
>> Use local disks - never EBS, and if you really care about writing speeds
>> use SSDs.
>>
>> Other than that, Mike did an excellent write up on the subject :
>> http://www.elasticsearch.org/blog/performance-
>> considerations-elasticsearch-indexing/
>>
>> --
>>
>> Itamar Syn-Hershko
>> http://code972.com | @synhershko 
>> Freelance Developer & Consultant
>> Lucene.NET committer and PMC member
>>
>> On Thu, Feb 5, 2015 at 1:26 PM, Hagai T  wrote:
>>
>>> Hi Guys,
>>>
>>> We use ElasticSearch as our tracking system of our products in a dynamic
>>> to track performance.
>>> Searching of this data is being used by small group of users (10-12) in
>>> the company to measure performance.
>>>
>>> In our current environment, we see limit of 15,000 documents being
>>> inserted per second without the ability to scale.
>>>
>>> Some information on the current setup and  the flow:
>>>
>>> *- Tracking servers*
>>> 8 x Apache servers behind Amazon ELB which serve empty html files so it
>>> tracks the parameters given and writes it to Apache access log.
>>> on each server, we also have Logstash which configured to read this
>>> access log file and send data to Elasticsearch cluster.
>>>
>>> *- Elasticsearch Cluster: *
>>> 4 x r3.2xlarge (61.0 GB RAM, 8 cores) - contains one Elasticsearch
>>> process - 30GB Heap size
>>> 1 x r3.4xlarge (122.0 GB RAM, 16 cores) - contains two Elasticsearch
>>> processes each with 30GB Heap size.
>>>
>>>
>>> Additional information on the Cluster:
>>> https://gist.github.com/hagait/61e3fac181ff413a8b8c#file-gistfile1-txt
>>>
>>> Cluster health: https://gist.github.com/hagait/d4f16d8f7b724f85b0ee#
>>> file-gistfile1-txt
>>>
>>> Logstash Configuration: https://gist.github.com/
>>> hagait/23f4b2bc614a4c4acbb6
>>>
>>> Elasticsearch configuration: https://gist.github.com/hagait/
>>> ba3684048abe2f9219b8
>>>
>>> Thank you for the support!
>>> Regards,
>>> Hagai
>>>
>>>
>>>  --
>>> You received this message because you are subscribed to the Google
>>> Groups "elasticsearch" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to elasticsearc...@googlegroups.com.
>>> To view this discussion on the web visit https://groups.google.com/d/
>>> msgid/elasticsearch/9c5352ed-010d-4080-9794-b8dc0c2a5370%
>>> 40googlegroups.com
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/318b46ed-626f-4975-a417-c99ada9c30fe%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAHTr4Zv0toyUEhc%2BynpkH%3DUxf9GVLuvnKdPSGUQcnG1grwpbCw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: Performance Limitation with ELK stack

2015-02-05 Thread Hagai T

Hi Itamar, thank you for the reply.

This is 15k inserts totally and not for one host in the cluster.
Yes, we have 15 sharding whiting one index. shards are spreaded on the 
nodes equally (automatically by Elasticsearch cluster).
We currently use general purpose SSD and not Ephemeral storage.

In addition, I see a lot of thread pool bulk rejections from the 
Elasticsearch side.


On Thursday, February 5, 2015 at 1:33:05 PM UTC+2, Itamar Syn-Hershko wrote:
>
> What is the question?...
>
> 15k inserts per sec per node is actually quite nice.
>
> Are your index sharded? If you write to one index only, you write to 
> maximum of x nodes where x is the number of shards of that index. Since 
> shards of the same index can co-exist on one node, check if you are 
> spanning writes.
>
> Use local disks - never EBS, and if you really care about writing speeds 
> use SSDs.
>
> Other than that, Mike did an excellent write up on the subject : 
> http://www.elasticsearch.org/blog/performance-considerations-elasticsearch-indexing/
>
> --
>
> Itamar Syn-Hershko
> http://code972.com | @synhershko 
> Freelance Developer & Consultant
> Lucene.NET committer and PMC member
>
> On Thu, Feb 5, 2015 at 1:26 PM, Hagai T > 
> wrote:
>
>> Hi Guys,
>>
>> We use ElasticSearch as our tracking system of our products in a dynamic 
>> to track performance.
>> Searching of this data is being used by small group of users (10-12) in 
>> the company to measure performance.
>>
>> In our current environment, we see limit of 15,000 documents being 
>> inserted per second without the ability to scale.
>>
>> Some information on the current setup and  the flow:
>>
>> *- Tracking servers*
>> 8 x Apache servers behind Amazon ELB which serve empty html files so it 
>> tracks the parameters given and writes it to Apache access log.
>> on each server, we also have Logstash which configured to read this 
>> access log file and send data to Elasticsearch cluster.
>>
>> *- Elasticsearch Cluster: *
>> 4 x r3.2xlarge (61.0 GB RAM, 8 cores) - contains one Elasticsearch 
>> process - 30GB Heap size
>> 1 x r3.4xlarge (122.0 GB RAM, 16 cores) - contains two Elasticsearch 
>> processes each with 30GB Heap size.
>>  
>>
>> Additional information on the Cluster: 
>> https://gist.github.com/hagait/61e3fac181ff413a8b8c#file-gistfile1-txt
>>
>> Cluster health: 
>> https://gist.github.com/hagait/d4f16d8f7b724f85b0ee#file-gistfile1-txt
>>
>> Logstash Configuration: 
>> https://gist.github.com/hagait/23f4b2bc614a4c4acbb6
>>
>> Elasticsearch configuration: 
>> https://gist.github.com/hagait/ba3684048abe2f9219b8
>>
>> Thank you for the support!
>> Regards,
>> Hagai
>>
>>
>>  -- 
>> You received this message because you are subscribed to the Google Groups 
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to elasticsearc...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/elasticsearch/9c5352ed-010d-4080-9794-b8dc0c2a5370%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/318b46ed-626f-4975-a417-c99ada9c30fe%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Performance Limitation with ELK stack

2015-02-05 Thread Itamar Syn-Hershko

What is the question?...

15k inserts per sec per node is actually quite nice.

Are your index sharded? If you write to one index only, you write to
maximum of x nodes where x is the number of shards of that index. Since
shards of the same index can co-exist on one node, check if you are
spanning writes.

Use local disks - never EBS, and if you really care about writing speeds
use SSDs.

Other than that, Mike did an excellent write up on the subject :
http://www.elasticsearch.org/blog/performance-considerations-elasticsearch-indexing/

--

Itamar Syn-Hershko
http://code972.com | @synhershko 
Freelance Developer & Consultant
Lucene.NET committer and PMC member

On Thu, Feb 5, 2015 at 1:26 PM, Hagai T  wrote:

> Hi Guys,
>
> We use ElasticSearch as our tracking system of our products in a dynamic
> to track performance.
> Searching of this data is being used by small group of users (10-12) in
> the company to measure performance.
>
> In our current environment, we see limit of 15,000 documents being
> inserted per second without the ability to scale.
>
> Some information on the current setup and  the flow:
>
> *- Tracking servers*
> 8 x Apache servers behind Amazon ELB which serve empty html files so it
> tracks the parameters given and writes it to Apache access log.
> on each server, we also have Logstash which configured to read this access
> log file and send data to Elasticsearch cluster.
>
> *- Elasticsearch Cluster: *
> 4 x r3.2xlarge (61.0 GB RAM, 8 cores) - contains one Elasticsearch
> process - 30GB Heap size
> 1 x r3.4xlarge (122.0 GB RAM, 16 cores) - contains two Elasticsearch
> processes each with 30GB Heap size.
>
>
> Additional information on the Cluster:
> https://gist.github.com/hagait/61e3fac181ff413a8b8c#file-gistfile1-txt
>
> Cluster health:
> https://gist.github.com/hagait/d4f16d8f7b724f85b0ee#file-gistfile1-txt
>
> Logstash Configuration:
> https://gist.github.com/hagait/23f4b2bc614a4c4acbb6
>
> Elasticsearch configuration:
> https://gist.github.com/hagait/ba3684048abe2f9219b8
>
> Thank you for the support!
> Regards,
> Hagai
>
>
>  --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elasticsearch+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/9c5352ed-010d-4080-9794-b8dc0c2a5370%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuuedagMgkbe4JoUGWg2DT1pFMkmdRKS3Rp4f4mwq6ntg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Performance Limitation with ELK stack

2015-02-05 Thread Hagai T

Hi Guys,

We use ElasticSearch as our tracking system of our products in a dynamic to
track performance.
Searching of this data is being used by small group of users (10-12) in the
company to measure performance.

In our current environment, we see limit of 15,000 documents being inserted
per second without the ability to scale.

Some information on the current setup and the flow:

*- Tracking servers*
8 x Apache servers behind Amazon ELB which serve empty html files so it
tracks the parameters given and writes it to Apache access log.
on each server, we also have Logstash which configured to read this access
log file and send data to Elasticsearch cluster.

*- Elasticsearch Cluster: *
4 x r3.2xlarge (61.0 GB RAM, 8 cores) - contains one Elasticsearch process
- 30GB Heap size
1 x r3.4xlarge (122.0 GB RAM, 16 cores) - contains two Elasticsearch
processes each with 30GB Heap size.

Additional information on the Cluster:
https://gist.github.com/hagait/61e3fac181ff413a8b8c#file-gistfile1-txt

Cluster health:
https://gist.github.com/hagait/d4f16d8f7b724f85b0ee#file-gistfile1-txt

Logstash Configuration: https://gist.github.com/hagait/23f4b2bc614a4c4acbb6

Elasticsearch
configuration: https://gist.github.com/hagait/ba3684048abe2f9219b8

Thank you for the support!
Regards,
Hagai

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to elasticsearch+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/9c5352ed-010d-4080-9794-b8dc0c2a5370%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Performance Limitation with ELK stack

Re: Performance Limitation with ELK stack

Re: Performance Limitation with ELK stack

Re: Performance Limitation with ELK stack

Re: Performance Limitation with ELK stack

Re: Performance Limitation with ELK stack

Performance Limitation with ELK stack

7 matches

Site Navigation

Mail list logo

Footer information