Simon, The key issue is that the procedure below has only been performed a few times, and only by people outside of Basho. No complaint has surfaced, but that does not mean the procedure is safe or thoroughly tested.
We have developed a plan for addressing the situation safely, but it is several weeks away. That said, your three point plan below is the best known available today. Matthew On Sep 26, 2013, at 5:09 PM, Simon Effenberg <seffenb...@team.mobile.de> wrote: > Thanks Matthew.. > > but to clarify.. > > 1. remove big .sst files > 2. trigger repair through > https://gist.github.com/gburd/b88aee6da7fee81dc036 > 3. wait for AAE or do a repair by hand with > http://docs.basho.com/riak/1.3.1/cookbooks/Repairing-KV-Indexes/ to > get all missing (because deleted) keys? > > Is this true what I'm saying? > > Cheers > Simon > > On Thu, 26 Sep 2013 15:06:10 +0200 > Matthew Von-Maszewski <matth...@basho.com> wrote: > >> Simon, >> >> I queried Basho's Client Services group. They use the following as the >> guideline for leveldb vnode (database) repair: >> >> https://gist.github.com/gburd/b88aee6da7fee81dc036 >> >> This repair sequence is different than the one you are quoting. >> >> >> You do not need to manually delete the MANIFEST file. The repair sequence >> will do that anyway. >> >> Matthew >> >> >> On Sep 26, 2013, at 7:04 AM, Simon Effenberg <seffenb...@team.mobile.de> >> wrote: >> >>> Are we talking about >>> http://docs.basho.com/riak/1.3.1/cookbooks/Repairing-KV-Indexes/ ? >>> >>> and if yes.. is this also doing a repair if data is missing? So after >>> executing the "repair" on one node.. will it have afterwards: >>> >>> * all keys it should have (like if AAE is fixing stuff through Read >>> Repair) >>> * all 2i indices it should have (like the documentation suggest) >>> >>> So does it mean a "compaction" is triggered by doing: >>> >>> * remove big sst files >>> * trigger this REPAIR type "Repairing-KV-Indexes"? >>> >>> will this automatically fix the MANIFESTS file? >>> >>> Cheers >>> Simon >>> >>> >>> On Thu, 26 Sep 2013 12:53:40 +0200 >>> Simon Effenberg <seffenb...@team.mobile.de> wrote: >>> >>>> Thanks Matthew, >>>> >>>> but one thing I didn't get. If I delete a sst file.. should I delete >>>> (by hand) the MANIFEST file to trigger a repair or is it done >>>> automatically within Riak if it detects that a sst file which is >>>> referenced in the MANIFEST is missing? >>>> >>>> Cheers >>>> Simon >>>> >>>> On Wed, 25 Sep 2013 09:58:51 -0400 >>>> Matthew Von-Maszewski <matth...@basho.com> wrote: >>>> >>>>> Simon, >>>>> >>>>> I want to point out again that I do not like my answer … but currently >>>>> lack a better one. The manual delete of .sst files should be treated as >>>>> a last resort. Only do it if you really, really cannot wait for normal >>>>> compactions to purge the old data. >>>>> >>>>> To your question: >>>>> >>>>> leveldb keeps a list of .sst table files in its MANIFEST file. If the >>>>> .sst table files are deleted manually, the MANIFEST is no longer >>>>> accurate. An inaccurate MANIFEST file leads to leveldb internal errors >>>>> and sometimes infinite loops. >>>>> >>>>> The repair process erases the current MANIFEST file and builds a new one >>>>> based upon the files actually found on disk. >>>>> >>>>> WARNING: repair takes a really, really long time in Riak 1.2.0 … it gets >>>>> better in 1.2.1 … and improves to something quite reasonable by 1.3 and >>>>> 1.4. >>>>> >>>>> A discussion about repair can be found here: >>>>> >>>>> https://github.com/basho/leveldb/wiki/repair-notes >>>>> >>>>> Matthew >>>>> >>>>> >>>>> On Sep 25, 2013, at 9:38 AM, Simon Effenberg <seffenb...@team.mobile.de> >>>>> wrote: >>>>> >>>>>> On Wed, 25 Sep 2013 09:15:33 -0400 >>>>>> Matthew Von-Maszewski <matth...@basho.com> wrote: >>>>>> >>>>>>> - run Riak repair on the vnode so that leveldb can create a MANIFEST >>>>>>> that matches the files remaining. >>>>>> >>>>>> what do you mean with this? Wait for AAE? Request each key to trigger >>>>>> read repair or do I miss something? >>>>>> >>>>>> We are in a similar situation.. only sst_4 exists in our situation but >>>>>> we also delete old stuff regularly.. >>>>>> >>>>>> Cheers >>>>>> Simon >>>>>> >>>>>> _______________________________________________ >>>>>> riak-users mailing list >>>>>> riak-users@lists.basho.com >>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>>>> >>>> >>>> >>>> -- >>>> Simon Effenberg | Site Ops Engineer | mobile.international GmbH >>>> Fon: + 49-(0)30-8109 - 7173 >>>> Fax: + 49-(0)30-8109 - 7131 >>>> >>>> Mail: seffenb...@team.mobile.de >>>> Web: www.mobile.de >>>> >>>> Marktplatz 1 | 14532 Europarc Dreilinden | Germany >>>> >>>> >>>> Geschäftsführer: Malte Krüger >>>> HRB Nr.: 18517 P, Amtsgericht Potsdam >>>> Sitz der Gesellschaft: Kleinmachnow >>>> >>>> _______________________________________________ >>>> riak-users mailing list >>>> riak-users@lists.basho.com >>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>> >>> >>> -- >>> Simon Effenberg | Site Ops Engineer | mobile.international GmbH >>> Fon: + 49-(0)30-8109 - 7173 >>> Fax: + 49-(0)30-8109 - 7131 >>> >>> Mail: seffenb...@team.mobile.de >>> Web: www.mobile.de >>> >>> Marktplatz 1 | 14532 Europarc Dreilinden | Germany >>> >>> >>> Geschäftsführer: Malte Krüger >>> HRB Nr.: 18517 P, Amtsgericht Potsdam >>> Sitz der Gesellschaft: Kleinmachnow >>> >>> _______________________________________________ >>> riak-users mailing list >>> riak-users@lists.basho.com >>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >> > > > -- > Simon Effenberg | Site Ops Engineer | mobile.international GmbH > Fon: + 49-(0)30-8109 - 7173 > Fax: + 49-(0)30-8109 - 7131 > > Mail: seffenb...@team.mobile.de > Web: www.mobile.de > > Marktplatz 1 | 14532 Europarc Dreilinden | Germany > > > Geschäftsführer: Malte Krüger > HRB Nr.: 18517 P, Amtsgericht Potsdam > Sitz der Gesellschaft: Kleinmachnow _______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
