As an aside; is there a good write-up somewhere about charm unit testing. I'd like to do this but I'm not sure how to do this. I am completely new to unit testing so I'm having a hard time to see how a good unittest for a Charm would look like and what exactly should be tested.
2016-03-17 1:52 GMT+01:00 Marco Ceppi <marco.ce...@canonical.com>: > Hello everyone! > > This is an email I've been meaning to write for a while, and have > rewritten a few times now. With 2.0 on the horizon and the charm ecosystem > rapidly growing, I couldn't keep the idea to myself any longer. > > # tl;dr: > > We should stop writing Amulet tests in charms and instead only write them > Bundles and force charms to do unit-testing (when possible) and promote > that all charms be included in bundles in the store. > > # Problem > > Without making this a novel, charm-testing and amulet started before > bundles were even a construct in Juju with a spec written before Juju 1.0. > Since then, many new comers to the ecosystem have remarked how odd it is to > be writing deployment validations at the charm level. Indeed, as years have > gone by and new tools have sprung up it's become clear that; having an > author try to model all the permutations of a charms deployment and do the > physical deploys at that charm level are tedious and incomplete at best. > > With the explosion of layers and improvements to uniting test in charms at > that component level, I feel that continuing to create these bespoke > "bundles" via amulet in a single charm will not be a robust solution going > forward. As we sprint closer to Juju 2.0 we're seeing a higher demand for > assurance of working scenarios, and a sharp focus on quality at every > level. As such I'd like to propose the following policy changes: > > - All bundles must have tests before promulgation to the store > - All charms need to have comprehensive tests (unit or amulet) > - All charms should be included in a bundle > > I'll break down my reasoning and examples in the following sections: > > # All bundles must have tests before promulgation to the store > > Writing bundle tests with Amulet is actually a more compelling story today > than writing an Amulet test case for a charm. As an example, there's a new > ELK stack bundle being produced, here's what the test for that bundle looks > like: > https://github.com/juju-solutions/bundle-elk-stack/blob/master/tests/10-test-bundle > > This makes a lot of sense because it's asserting that the bundle is > working as expected by the Author who put the bundle together. It's also > loading the bundle.yaml as the deployment spec meaning as the bundle > evolves the tests will make sure they continue to run as expected. Also, > this could potentially be used in future smoke tests for charms being > updated if a CI process swaps out, say elasticsearch, for a newer version > of a charm being reviewed. We can assert that both the unittests in > elasticsearch work and it operates properly in an existing real world > solution a la the bundle. > > Additional examples: > - > https://github.com/juju-solutions/bundle-realtime-syslog-analytics/blob/master/tests/01-bundle.py > - > https://github.com/juju-solutions/bundle-apache-core-batch-processing/blob/master/tests/01-bundle.py > > # All charms need to have comprehensive tests (unit or amulet) > > This is just a clarification and more strongly typed policy change that > require charms have (preferred) unit tests or, if not applicable, then an > Amulet test. Bash doesn't really allow for unittesting, so in those > scenarios, Amulet tests would function as a valid testing case. > > There are also some charms which will not make sense as a bundle. One > example is the recently promulgated Fiche charm: > http://bazaar.launchpad.net/~charmers/charms/trusty/fiche/trunk/view/head:/tests/10-deploy > It's > a standalone pastebin, but it's an awesome service that provides deployment > validation with an Amulet test. The test stands up the charm, exercises > configuration, and validates the service responds in an expected way. For > scenarios where a charm does not have a bundle an Amulet test would be > required. > > Any charm that currently includes an Amulet test is welcome to continue > keeping such a test. > > # All charms should be included in a bundle > > This last one is to underscore that charms need to serve a purpose. This > policy is written as not an absolute, but instead a strongly worded > suggestion as there are always charms that are exceptions to the rules. One > such example is the aforementioned Fiche charm which as a bundle would not > make as much sense, but is still a purposeful charm. > > That being said, most users coming to consume Juju are looking to solve a > problem. Bundles underscore solutions to problems that people can consume, > and get started quicker. > > As such, when new applications are charmed a test of "is this application > something that serves a clear purpose" having a bundle submitted alongside > the charm validates that claim and provides users a way to immediately get > started with a solution. > > # Conclusion > > These policy changes, once accepted, will be targeted at all charms and > bundles in Xenial as well as any new charm submitted after policy > acceptance date for trusty, and finally any charm currently under review > will be encouraged to adhere to the new policy but won't be required. > > # Action items > > I'm seeking feedback on this concept and welcome suggestions for > improvements, questions, dissenting opinions, and any other remarks as well > as votes from ~charmers and feedback from the community at large. > > Thanks, > Marco Ceppi > > -- > Juju mailing list > Juju@lists.ubuntu.com > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/juju > >
-- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
