Tool for interactive fault injection testing

2012-07-02 Thread Andrei Savu
Hi guys, As part of my MSc. project I have spent some time working on a tool for fault injection testing for Apache ZooKeeper based on jboss Byteman and Apache Whirr. You can find the code on Github: https://github.com/andreisavu/zookeeper-tester Do you think this can be an useful addition to c

Re: Tool for interactive fault injection testing

2012-07-02 Thread Patrick Hunt
Sounds interesting but it's not clear to me from the provided docs what it does and what am I expected to do? (canned tests or a framework for me to use). Have you been able to find any issues using this? Patrick On Mon, Jul 2, 2012 at 3:15 AM, Andrei Savu wrote: > Hi guys, > > As part of my MSc

Re: Tool for interactive fault injection testing

2012-07-02 Thread Andrei Savu
I was unable to find any issues so far. It seems like ZooKeeper does a great job at handling network failures. This tool is deploying a ZooKeeper cluster on a cloud provider using Whirr together with Byteman [1] (attached to the JVM). Faults are injected by using Byteman rules. See this tutorial

Re: Tool for interactive fault injection testing

2012-07-02 Thread Flavio Junqueira
Sounds like great stuff, Andrei. Do you have a description of the faults you have injected I can access? -Flavio On Jul 2, 2012, at 10:14 PM, Andrei Savu wrote: > I was unable to find any issues so far. It seems like ZooKeeper does a > great job at > handling network failures. > > This tool is

Re: Tool for interactive fault injection testing

2012-07-02 Thread Andrei Savu
Thanks Flavio! This is the rule I'm using for the demo: RULE NIO Server readPayload fails CLASS org.apache.zookeeper.server.NIOServerCnxn METHOD readPayload HELPER RandomHelper AT ENTRY IF nextInt(100) < 10 DO throw new IOException("Injected by byteman"); ENDRULE See: https://github.com/andreis

Re: Tool for interactive fault injection testing

2012-07-03 Thread Flavio Junqueira
Hi Andrei, Here are some thoughts about fault injection. Broadly speaking, I think we can classify the faults into server and channel. Server faults can mess with the internals of a server, and can cause it to crash, delay a response, or corrupt data. We have only limited support to tolerate dat

Re: Tool for interactive fault injection testing

2012-07-06 Thread Patrick Hunt
Andrei, FYI I've passed this on to my ZK QA lead and asked him to followup with you. I'm hoping we can add this to our system testing infrastructure. Regards, Patrick On Tue, Jul 3, 2012 at 2:14 PM, Flavio Junqueira wrote: > Hi Andrei, Here are some thoughts about fault injection. Broadly speak