Intentionally introducing failures into Juju

2014-08-13 Thread Menno Smits
There's been some discussion recently about adding some feature to Juju to allow developers or CI tests to intentionally trigger otherwise hard to induce failures in specific parts of Juju. The idea is that sometimes we need some kind of failure to happen in a CI test or when manually testing but t

Re: Intentionally introducing failures into Juju

2014-08-13 Thread Gustavo Niemeyer
That's a nice direction, Menno. The main thing that comes to mind is that it sounds quite inconvenient to turn the feature on. It may sound otherwise because it's so easy to drop files at arbitrary places in our local machines, but when dealing with a distributed system that knows how to spawn its

Re: Intentionally introducing failures into Juju

2014-08-13 Thread Gustavo Niemeyer
Ah, and one more thing: when developing the chaos-injection mechanism in the mgo/txn package, I also added both a "chance" parameter for either killing or slowing down a given breakpoint. It sounds like it would be useful for juju's mechanism too. If you kill every time, it's hard to tell whether t

Re: Intentionally introducing failures into Juju

2014-08-13 Thread Wayne Witzel
Not much to add except to say I really like this work and I think it is going to really help us make Juju much better when encountering failures. I also like the idea of providing easy access to triggering failures through CLI commands. On Wed, Aug 13, 2014 at 10:29 AM, Gustavo Niemeyer < gustav

Re: Intentionally introducing failures into Juju

2014-08-13 Thread Menno Smits
I like the idea being able to trigger failures using the juju command line. I'm undecided about how the need to fail should be stored. An obvious location would be in a new collection managed by state, or even as a field on existing state objects and documents. The downside of this approach is tha

Re: Intentionally introducing failures into Juju

2014-08-13 Thread Menno Smits
I like the idea of being able to trigger failures stochastically. I'll integrate this into whatever we settle on for Juju's failure injection. On 14 August 2014 02:29, Gustavo Niemeyer wrote: > Ah, and one more thing: when developing the chaos-injection mechanism > in the mgo/txn package, I als

Re: Intentionally introducing failures into Juju

2014-08-13 Thread Stuart Bishop
On 14 August 2014 07:31, Menno Smits wrote: > I like the idea being able to trigger failures using the juju command line. > > I'm undecided about how the need to fail should be stored. An obvious > location would be in a new collection managed by state, or even as a field > on existing state objec

Re: Intentionally introducing failures into Juju

2014-08-14 Thread Gustavo Niemeyer
On Thu, Aug 14, 2014 at 3:42 AM, Stuart Bishop wrote: > Further to just injecting failures, I'm interested in controlling when > and the order hooks can run. A sort of manual mode, which could be > driven by a test harness such as Amulet. This sounds quite heavyweight and intrusive. Introducing d

Re: Intentionally introducing failures into Juju

2014-08-16 Thread Stuart Bishop
On 14 August 2014 20:39, Gustavo Niemeyer wrote: > On Thu, Aug 14, 2014 at 3:42 AM, Stuart Bishop > wrote: >> Further to just injecting failures, I'm interested in controlling when >> and the order hooks can run. A sort of manual mode, which could be >> driven by a test harness such as Amulet. >