Hi Igor We are using elasticsearch 1.1.1. Currently we are keeping all snapshots that we make in s3, we haven't yet decided on an archive strategy/solution. So at the moment we have 131 snapshots in the s3 bucket. So we have about 112 new indices a day.
I'll explain our set up a bit it may well be that we should be doing something different. We are grabbing application logs from lots of different apps and putting them into elasticsearch. We are using flume to do this. So similar to logstash setup. The one major difference is that we are creating an index for each application. So if we have 100 apps we will create 100 indices each day one for each new app. Dip On Thursday, May 8, 2014 2:58:22 PM UTC+1, Igor Motov wrote: > > Hi Dipesh, > > I have a few questions. Are you still on S3? Which version of > elasticsearch are you using? How many snapshots do you currently keep in > S3? How fast is your index growing over time? > > Igor > > On Wednesday, May 7, 2014 6:58:05 AM UTC-4, Dipesh Patel wrote: >> >> Hi >> >> We've noticed recently that our snapshot durations are increasing over >> time. Our rate of flow of data going into elasticsearch has remained fairly >> constant. Though we do create new indices everyday ( though this is a fixed >> number that doesn't vary from day to day ). We are currently snapshoting, >> or trying to snapshot every hour. However with the snapshots taking a >> progressively longer time this is proving difficult. >> >> Here's some stats showing our time to finish: >> >> Name >> Duration ( milli ) >> snapshot_2014-05-01_01:30:00 4497010 snapshot_2014-05-01_03:30:00 >> 4513037 snapshot_2014-05-01_05:30:00 4770288 >> snapshot_2014-05-01_07:30:00 5413361 snapshot_2014-05-01_11:30:00 >> 6978384 snapshot_2014-05-01_13:30:00 6907554 >> snapshot_2014-05-01_15:30:00 7388500 >> This is just the tail end originally the snapshots were only taking 7-8 >> mins to run, they've just been getting progressively longer. >> >> Any help to debug etc appreciated. >> >> Dip >> > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/441319d1-a521-4a6b-9671-de541d129029%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
