Re: [DISCUSS] Fault injection tests for Kafka

YES! One of my goal for the fault-injection in our system tests is that whoever fixes the issue will also add tests to make sure it stays fixed. On Wed, Oct 5, 2016 at 11:33 AM, Tom Crayford wrote: > I did some stuff like this recently with simple calls to `tc` (samples

Re: [DISCUSS] Fault injection tests for Kafka

I did some stuff like this recently with simple calls to `tc` (samples that I used were in the README for https://github.com/tylertreat/comcast). The only notable bug I found so far is that if you cut all the kafka nodes entirely off from zookeeper for say, 60 seconds, then reconnect them, the

Re: [DISCUSS] Fault injection tests for Kafka

Yeah, totally agree on discussing what we want to test first and implement anything later :) Its just that whenever I have this discussion Jepsen came up, so I was curious what was driving the interest and whether the specific framework is important to the community. On Tue, Oct 4, 2016 at 5:46

Re: [DISCUSS] Fault injection tests for Kafka

for "small" failures (local failures on a single node, like socket disconnection, disk read errors, out of memory etc) I've used byteman before - http://byteman.jboss.org/ On Tue, Oct 4, 2016 at 5:46 PM, Joel Koshy wrote: > Hi Gwen, > > I've also seen suggestions of using

Re: [DISCUSS] Fault injection tests for Kafka

Hi Gwen, I've also seen suggestions of using Jepsen for fault injection, but > I'm not familiar with this framework. > > What do you guys think? Write our own failure injection? or write > Kafka tests in Jepsen? > This would definitely add a lot of value and save a lot on release validation