Goldman Sachs gave a talk about how they're using Elasticsearch to index 5TB of log data per day. I can't find the video of the talk, but from a blogpost about it:
Next was Indy Tharmakumar from our hosts Goldman Sachs<http://www.goldmansachs.com/>, showing how his team have built powerful support systems using ElasticSearch to index log data. Using 32 1 core CPU instances the system they have built can store 1.2 billion log lines with a throughput up to 40,000 messages a second (the systems monitored produce 5TB of log data every day). Log data is queued up in Redis <http://redis.io/>, distributed to many Logstash <http://logstash.net/> processes, indexed by Elasticsearch with a Kibana <http://rashidkpc.github.io/Kibana/> front end. They learned that Logstash can be particularly CPU intensive but Elasticsearch itself scales extremely well. Future plans include considering Apache Kafka<http://kafka.apache.org/> as a data backbone. On 19 April 2014 06:46, 熊贻青 <xiong.jag...@gmail.com> wrote: > We have a cluster with 10 nodes, 48g heap for each ES process. The total > indexing rate is about 25000 doc per second, about 20 indices actively > receiving new data. I'm really courious to compare and evaluate the > indexing performance numers. > > Thanks! > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to elasticsearch+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/CAP0hgQ34ZwY6Or0PUFZn_Ciu_iyZZJjyXfz%3DNBu64Ge9uN3hxQ%40mail.gmail.com<https://groups.google.com/d/msgid/elasticsearch/CAP0hgQ34ZwY6Or0PUFZn_Ciu_iyZZJjyXfz%3DNBu64Ge9uN3hxQ%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPt3XKS6AnOVKwNszU-SFYmyGUpk57U_kz8iXQssGk%3DX81KMiA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
