Whatson shows this very well: https://wikitech.wikimedia.org/wiki/File:Whatson_out_of_disk.png
Other points of interest: 1. We're using auto_expand_replicas. 2. The logs are totally clean. Nik On Thu, Aug 21, 2014 at 10:35 AM, Nikolas Everett <nik9...@gmail.com> wrote: > This gist shows the error in action: > https://gist.github.com/nik9000/3acdb38052dba3fbc5a0 > Total - free on disk is 479163707392 > But used is 238902736642 > Meaning about 50% of used space isn't accounted for. > But everything on that partition is in elasticsearch's directory: > manybubbles@elastic1001:/var/lib/elasticsearch/production-search-eqiad/nodes/0/indices$ > du -h | tail -n1 > 447G . > > Its like when we did the upgrade some files weren't deleted when they were > no longer in use. > > > On Thu, Aug 21, 2014 at 10:24 AM, Nikolas Everett <nik9...@gmail.com> > wrote: > >> Hi Lee! Thanks for responding. Ok, here goes: >> >> Version: 1.2.1->1.3.2 >> curl 'localhost:9200/_cat/health?v: >> epoch timestamp cluster status node.total node.data >> shards pri relo init unassign >> 1408630877 14:21:17 production-search-eqiad green 17 >> 17 6050 2017 0 0 0 >> >> curl 'localhost:9200/_cat/shards?v': >> https://gist.github.com/nik9000/815f3e39a3673e6f48ac >> >> I'll keep digging into it now that I'm properly awake as well. >> >> Nik >> >> >> On Thu, Aug 21, 2014 at 9:36 AM, Lee Hinman <matthew.hin...@gmail.com> >> wrote: >> >>> On Thursday, August 21, 2014 2:44:19 PM UTC+2, Nikolas Everett wrote: >>>> >>>> I started a rolling restart yesterday but has add to stop because the >>>> disks were filling up oddly. It looks like when the bode comes up it no >>>> longer deletes shards it can't use. >>>> >>>> Elasticsearch reports that the disk is nearly full but that it isn't >>>> using most of the space. When I look myself the disk is mostly full and >>>> most of the space is taken up by shards. >>>> >>>> I'm not clear where to go from here though. Find the files >>>> elasticsearch doesn't have open and delete them? >>>> >>> >>> Hi Nikolas, >>> >>> Can you provide the output of `curl 'localhost:9200/_cat/shards?v'` and >>> `curl 'localhost:9200/_cat/health?v'`? Also, can you describe your cluster >>> topology and what the current disk usages are for all nodes across the >>> cluster? >>> >>> Additionally, what version of ES are you using before and after the >>> upgrade? >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "elasticsearch" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to elasticsearch+unsubscr...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elasticsearch/4bcb3a29-7fb8-4f09-80fd-83dbd7d81c47%40googlegroups.com >>> <https://groups.google.com/d/msgid/elasticsearch/4bcb3a29-7fb8-4f09-80fd-83dbd7d81c47%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3-%3D2GjBejyU5szEVj%2BqY_%3DxgGybe9nLhHcZ418uVwFdA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
