I'm planning to start working on a nightly build that can spin up a mini-cluster, load some data into it, do some actions to bring the cluster into an undesirable state that hbck2 can fix and then invoke the hbck2 to see if things work well.
Plan is to start small with one of the hbck2 commands and remaining ones can be added incrementally. As of now I would like to start with making sure the job uses one of the hbase versions (probably 2.1.x), we can discuss about the need to run the job against all the present hbase versions/taking in a bunch of hbase versions as input and running against them/or just a single version. The job script would be located in our operator-tools repo. Let me start digging into creation of a nightly job in the operator-tools (I don't think we have any as of now). Will create a tracking jira for this. Further discussions regarding this can be delegated to the jira if deemed more convenient. -Sakthi On Mon, Oct 7, 2019 at 11:51 AM Andrew Purtell <apurt...@apache.org> wrote: > > We need something different from the chaos monkeys. in this case we're > not trying to peturb the cluster in ways we think it should handle; we're > setting up a state we already know requires an outside tool. > > Not sure this really falls outside the framework. Add an action that > invokes hbck. Then, add a policy that schedules the hbck invocation as part > of the schedule. That policy would also include the destructive actions > breaking things in a way the tool needs to fix (or HBase can handle > intrinsically...) > > On Tue, Oct 1, 2019 at 2:11 AM Sean Busbey <bus...@apache.org> wrote: > > > I was chatting with Sakthi about automating some testing of hbck2 > commands. > > Nothing too fancy, I just want some assurance that they ought to work. > > > > This got us talking about how we might purposefully break a cluster to > meet > > a set of symptoms that hbck2 knows how to correct. We need something > > different from the chaos monkeys. in this case we're not trying to peturb > > the cluster in ways we think it should handle; we're setting up a state > we > > already know requires an outside tool. > > > > Where should this kind of tooling live? Main repo next to the monkeys? > > Alongside hbck2 in operator tools? Somewhere else entirely? > > > > > -- > Best regards, > Andrew > > Words like orphans lost among the crosstalk, meaning torn from truth's > decrepit hands > - A23, Crosstalk >
