I ran the compaction 3 times. Each time freeing ~ 1GB of disk space. Further compactions now return immediately showing no segmented compacted and no bytes compacted.
The docs for merge_index:compact/1 state that this function "Performs compaction of segments if needed". If this function decided that no compaction was needed does that mean that all the tombstones are gone? There is no more disk space that could possibly be freed? -- Jeremy On Fri, Nov 2, 2012 at 9:52 AM, Jeremy Raymond <jeraym...@gmail.com> wrote: > Here is the new listing of the segments: > https://gist.github.com/4001468. Some files changed and some didn't. > Not really sure how to interpret the differences. > > I'll run the repeated compaction calls and grab the directory listing > after each iteration and see if that frees any space. > > -- > Jeremy > > > On Thu, Nov 1, 2012 at 4:32 PM, Ryan Zezeski <rzeze...@basho.com> wrote: > > Jeremy, > > > > I was looking at the merge index code and I think the issue is that the > > method by which segments are chosen for compaction may be very slow to > get > > to the larger segments. > > > > 1. Merge Index only schedules merging when a buffer is rolled-over to a > > segment. This means there will _always_ be at least one small segment in > > the list of potential segments to merge. > > > > 2. To determine which segments to merge the mean of all segment sizes is > > taken. > > > > Over time the mean will skew left of the bulk of the distribution. This > > means most compactions will touch only recent, smaller segments and it > will > > take many iterations before one of the larger ones is included. To help > > verify this you could list all you segment sizes again and compare them > with > > the last run. My guess is you'll have about the same number of segments > but > > the smallest one will have grown a bit. It depends how much unique data > you > > re-indexed. > > > > Depending on the distribution of your segment sizes I think it might be > > possible to reclaim some of this space via repeated compaction calls. It > > turns out there is a way to manually invoke compaction. It's just not > easy > > to get too. Try running the following gist on one of your nodes > > https://gist.github.com/3996286. Try running merge_index:compact over > and > > over again and each time check for changes in the segment file sizes. > > > > > > -Z > > > > > > On Thu, Nov 1, 2012 at 11:25 AM, Jeremy Raymond <jeraym...@gmail.com> > wrote: > >> > >> I reindexed a bunch of items that are still in the search index but no > >> disk space was reclaimed. Is there any Riak console Erlang voodoo I > >> can do to convince Riak Search that now would be a good time to > >> compact the merge_index? > >> > >> -- > >> Jeremy > >> > >> > >> On Tue, Oct 30, 2012 at 4:26 PM, Jeremy Raymond <jeraym...@gmail.com> > >> wrote: > >> > I've posted the list of buffer files [1] and segment files [2]. > >> > > >> > The current data set I have in Riak is static, so no new items are > >> > being written. So this looks like the reason as to why compaction > >> > isn't happening since there is no time based trigger on the merge > >> > index. To get compaction to kick in, I should be able to to just > >> > reindex (by reading and rewriting) some of the existing items in > >> > buckets that are still indexed? Earlier today I upgraded to Riak 1.2 > >> > and ran a Search read repair [3] in an attempt to kick of compaction. > >> > Compaction didn't kick in, but instead disk consumption increased > >> > again. Should Search Repair trigger compaction or only writing objects > >> > to the KV store? > >> > > >> > [1]:https://gist.github.com/3982718 > >> > [2]:https://gist.github.com/3982730 > >> > > >> > [3]: > http://docs.basho.com/riak/latest/cookbooks/Repairing-Search-Indexes/#Running-a > >> > Repair > >> > > >> > -- > >> > Jeremy > >> > > >> > > >> > On Tue, Oct 30, 2012 at 3:47 PM, Ryan Zezeski <rzeze...@basho.com> > >> > wrote: > >> >> find /var/lib/riak/merge_index -name 'buffer.*' | xargs ls -lah > >> >> > >> >> find /var/lib/riak/merge_index -name 'segment.*' | xargs ls -lah > > > > >
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
