There's not a lot you can do here unless you want to start uploading snapshots to S3, or something else that is not on your NAS. ES does compress by default and we are working on using a better algorithm for future releases which will help, but there's no ETA for that.
On 16 March 2015 at 17:29, David Reagan <jer...@gmail.com> wrote: > So, I haven't figured out the right search terms to find the answer via > Google yet, I've read a lot of the docs on the subject of Snapshot and > Restore without finding an answer, and I haven't had the time or resources > to test some of my own ideas. Hence, I'm posting this in the hopes that > someone who has already solved this problem will share. > > How do you run ES with limited data storage space? > > Basically, short of getting more space, what can I do to make the best use > of what I have, and still meet as many of my goals as possible? > > My setup is 4 data nodes. Due to lack of resources/money, they are all > thin provisioned VMs, and all my data has to be on NFS/SAN mounts. Storing > data on the actual VM's hard disk would negatively effect other VMs and > services. > > Our NFS SAN is also low on space. So I only have about 1.5TB to use. > Initially this seemed like plenty, but a couple weeks ago, ES started > complaining about running out of space. Usage on that mount was over 80%. > My snapshot repository had ballooned to over 700GB, and each node's data > mount point was around 150GB. > > Currently, I'm only using ES for logs. > > For day to day use, I should be fine with 1 month of open indices. Thus, > I've been keeping older indices closed already. So I can't really do much > more when it comes to closing indices. > > I also run the optimize command nightly on any logstash index older that a > couple days. > > I'd just delete the really old data, but I have use cases for data up to > 1.5 years old. Considering that snapshots of only a few months nearly used > up all my space, and how much space a month of logs is currently taking up, > I'm not sure how I can store that much data. > > So, in general, how would you solve my problem? I need to have immediate > access to 1 months worth of logs (via Kibana), be able to relatively > quickly access up to 6 months of logs (open closed indices?), and access up > to 1.5 years worth temporarily (restore snapshots to new cluster on my > desktop?) > > Would there be a way to move snapshots off of the NFS SAN to an external > hard drive? > > Should I tell logstash to send logs to a text file that get's logrotated > for a year and a half? Or does ES do a good enough job with compression > that gzipping wouldn't help? If it was just a text file, I could unzip it, > then tell Logstash to read the file into an ES cluster. > > ES already compresses stored indices by default, right? So there's nothing > I can do there? > > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to elasticsearch+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/b694768f-3c71-4b98-a18c-842c95809734%40googlegroups.com > <https://groups.google.com/d/msgid/elasticsearch/b694768f-3c71-4b98-a18c-842c95809734%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9DZhYKszmDRrssh%3DiNb6UAJ8EU6eHGPN-OPaORxmvM2w%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
