With the low volume of ingest, and the long duration of history, Id suggest you may want to trim back the number of shards per index from the default 5. Based on your 100 docs per day Id say 1 shard per day. If you combined this with the other suggestion to increase the duration of an index, then you might increase the number of shards, but maybe still not. Running an optimize once you have completed a time period is great advice if you can afford the overhead, sounds like one day at a time you should be able to, and that the overhead of not optimizing is costing you more when you snapshot.
And index is made of shards, a shard is made of lucene segments. Lucene segments are the actual files that you copy when you snapshot. As such the number of segments is multiplied by the number of shards per index and the number of indexes. Reducing the number of indexes by creating larger time periods will significantly reduce the number of segments. Reducing the number of shards per index will significantly reduce the number of segments. Optimizing the index will also consolidate many segments into a single segment. Based on the use of S3 should we assume you are using AWS EC2? What instance size? Your data volume seems very low so it seems concerning that you have such a large time period to snapshot, and points to a slow file system, or a significant number of segments (100 indexes, 5 shards per index, xx segments per shard, == many thousands of segments). What does your storage system look like? If you are using EC2 are you using the newer EBS volumes (SSD backed)? Some of the smaller instance size significantly limit prolonged EBS throughput, in my experience. On Wednesday, March 11, 2015 at 1:12:01 AM UTC-6, Magnus Bäck wrote: > > On Monday, March 09, 2015 at 20:29 CET, > Andy Nemzek <bitk...@gmail.com <javascript:>> wrote: > > > We've been using logstash for several months now and it creates a new > > index each day, so I imagine there are over 100 indexes at this point. > > Why create daily indexes if you only have a few hundred entries in each? > There's a constant overhead for each shard so you don't want more > indexes than you need. Seems like you'd be fine with montly indexes, > and then your snapshot problems would disappear too. > > > Elasticsearch is running on a single machine...I haven't done anything > > with shards, so the defaults must be in use. Haven't optimized old > > indexes. We're pretty much just running ELK out of the box. When you > > mention 'optimizing indexes', does this process combine indexes? > > No, but it can combine segments in a Lucene index (that make up > Elasticsearch indexes), and segments are what's being backed up. > So the more segments you have the the longer time snapshots are > going to take. > > > Do you know if these performance problems are typical when > > using ELK out of the box? > > 100 indexes on a single box should be okay but it depends on > the size of the JVM heap. > > -- > Magnus Bäck | Software Engineer, Development Tools > magnu...@sonymobile.com <javascript:> | Sony Mobile Communications > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7be4c805-b4f1-424d-b67b-2ad70e5da659%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
