On Thu, Mar 17, 2016 at 5:39 AM Tom Barber <t...@analytical-labs.com> wrote:
> I tend to agree with Ryan, I think the ideas are reasonably sound, > although I'm not sure about the "every charm should be part of a bundle" > policy, but I certainly don't think you should discourage testing at charm > level the encapsulation can be useful, and you can never have too many > tests! > Thanks for the feedback, I appreciate it. I want to clarify since I did a terrible job explaining that I don't wish to stop Amulet tests in charm but instead really focus on driving unittests in charms - and less charms but more Layers. This will allow for quicker iterations and faster dev/test cycles. > I'm > +0.5 for charms part of a bundle expectation > -1 for discouraging charm tests > Want to caveat again, charm tests would be one or the two (or both) [unittest, amulet] > My 2 cents. > Thanks! > > Tom > > -------------- > > Director Meteorite.bi - Saiku Analytics Founder > Tel: +44(0)5603641316 > > (Thanks to the Saiku community we reached our Kickstart > <http://kickstarter.com/projects/2117053714/saiku-reporting-interactive-report-designer/> > goal, but you can always help by sponsoring the project > <http://www.meteorite.bi/products/saiku/sponsorship>) > > On 17 March 2016 at 04:38, Ryan Beisner <ryan.beis...@canonical.com> > wrote: > >> Good evening, >> >> I really like the notion of a bundle possessing functional tests as an >> enhancement to test coverage. I agree with almost all of those ideas. :-) >> tldr; I would suggest that we consider bundle tests 'in addition to' and >> not 'as a replacement of' individual charm tests, because: >> >> >> *# Coverage and relevance* >> Any given charm may have many different modes of operation -- features >> which are enabled in some bundles but not in others. A bundle test will >> likely only exercise that charm in the context of its configuration as it >> pertains to that bundle. However, those who propose changes to the >> individual charm should know (via tests) if they've functionally broken the >> wider set of its knobs, bells and levers, which may be irrelevant to, or >> not testable in the bundle's amulet test due to its differing perspective. >> This opens potential functional test coverage gaps if we lean solely on the >> bundle for the test. >> >> There are numerous cases where a charm can shift personalities and use >> cases, but not always on-the-fly in an already-deployed model. In those >> cases, it may take a completely different and new deployment topology and >> configuration (bundle) to be able to exercise the relevant functional >> tests. Without integrated amulet tests within the charm, one would have to >> publish multiple bundles, each containing separate amulet tests. For >> low-dev-velocity charms, for simple charms, or for charms that aren't >> likely to be involved in complex workloads, this may be manageable. But I >> don't think we should discourage or stop looking for individual charm >> amulet tests even there. >> >> A charm's integrated amulet test can be both more focused and more >> expansive in what it exercises, as it can contain multiple deployment >> topologies and configurations (equivalent to cycling multiple unique >> bundles). For example: charm-xyz with and without SSL; or in HA and >> without HA; or IPv4 vs. IPv6; or IPv4 HA vs. IPv6 HA, multicast vs. >> unicast; [IPv6 + HA + SSL] vs [IPv4 + HA + SSL]; or mysql deploying mysql >> proper vs. mysql deploying a variant; and you can see the gist of the >> coverage explosion which translates to having a whole load of bundles to >> produce and maintain. >> >> >> *# Dev and test: cost, scale and velocity* >> Individual charm amulet tests are an important piece in testing large or >> complex models. I'll share some bits of what we do for OpenStack charms as >> an example. No bias. :-) >> >> Each of the OpenStack charms contain amulet test definitions. We lean >> heavily on those tests to deploy fractions of a full OpenStack bundle as >> the core of our CI development gate. With [27 charms] x [stable + dev] x >> [8 Ubuntu/OpenStack Release Combos], there are currently* ~432 *possible >> variations of amulet tests (derived bundles of fractional OpenStacks). A >> subset of those are executed in gate, depending on relevance to the >> developer's proposed change. This allows us to endure a high velocity of >> focused testing on development in these very active charms. Because the >> derived models are much smaller than the reference bundle, we can give >> developers rapid and automated feedback, plus they can iterate on >> development outside of our CI without having to be able to deploy a full >> OpenStack. >> >> That is not to say that we don't have acceptance and integration tests >> for full OpenStack bundles. We do that in the form of mojo specs which >> dynamically deploy any number of full OpenStack bundle topologies and >> configurations against multiple Ubuntu+OpenStack release combos, using >> either the dev or the stable set of OpenStack charms. It basically takes >> what I've described above for amulet and allows us to pivot entire bundles >> into different models automatically. There are currently *84* such >> OpenStack mojo specs with tests (bundle equivalents) >> >> Fear not, this is mostly accomplished with bundle inheritance, yaml foo, >> and shared test libraries. We're not actually maintaining ~*516 bundles*. >> But if we were to achieve the current level of coverage with bundles, >> that's approximately how many there would need to be. This includes the >> upcoming Xenial and Mitaka releases. Reduce by ~12% when Juno EOLs. Add >> 12% when we hit Newton B1, and so on. >> >> >> *# How I'd like to use the proposed ideas* >> There are some OpenStack reference bundles in the charm store. My >> suggested approach would be to continue to leverage individual charm amulet >> tests while adding functional tests to the existing charm store bundles. >> That would increase test coverage, and provide a mechanism to validate >> proposed changes to those specific bundles, such as to re-validate the >> bundles when charm versions are revved within them. >> >> >> To summarize, I am: >> >> -1 to stopping or discouraging individual charm amulet tests >> >> +1 for every charm containing amulet tests >> >> +1 for every charm containing unit tests >> >> +1 for every charm having amulet coverage in at least 1 bundle >> >> +1 for every bundle possessing amulet tests >> >> >> Also open to feedback, discussion, suggestions, kicks in the shin. >> >> Thanks for all the great tooling and thought leadership. We leverage the >> everloving *stuff* out of Amulet. >> >> Charm on! >> >> -- >> Ryan Beisner >> QA Engineer, Ubuntu OpenStack Engineering, Canonical, Ltd. >> irc:beisner gh/gerrit:ryan-beisner lp:~1chb1n >> >> >> >> On Wed, Mar 16, 2016 at 7:52 PM, Marco Ceppi <marco.ce...@canonical.com> >> wrote: >> >>> Hello everyone! >>> >>> This is an email I've been meaning to write for a while, and have >>> rewritten a few times now. With 2.0 on the horizon and the charm ecosystem >>> rapidly growing, I couldn't keep the idea to myself any longer. >>> >>> # tl;dr: >>> >>> We should stop writing Amulet tests in charms and instead only write >>> them Bundles and force charms to do unit-testing (when possible) and >>> promote that all charms be included in bundles in the store. >>> >>> # Problem >>> >>> Without making this a novel, charm-testing and amulet started before >>> bundles were even a construct in Juju with a spec written before Juju 1.0. >>> Since then, many new comers to the ecosystem have remarked how odd it is to >>> be writing deployment validations at the charm level. Indeed, as years have >>> gone by and new tools have sprung up it's become clear that; having an >>> author try to model all the permutations of a charms deployment and do the >>> physical deploys at that charm level are tedious and incomplete at best. >>> >>> With the explosion of layers and improvements to uniting test in charms >>> at that component level, I feel that continuing to create these bespoke >>> "bundles" via amulet in a single charm will not be a robust solution going >>> forward. As we sprint closer to Juju 2.0 we're seeing a higher demand for >>> assurance of working scenarios, and a sharp focus on quality at every >>> level. As such I'd like to propose the following policy changes: >>> >>> - All bundles must have tests before promulgation to the store >>> - All charms need to have comprehensive tests (unit or amulet) >>> - All charms should be included in a bundle >>> >>> I'll break down my reasoning and examples in the following sections: >>> >>> # All bundles must have tests before promulgation to the store >>> >>> Writing bundle tests with Amulet is actually a more compelling story >>> today than writing an Amulet test case for a charm. As an example, there's >>> a new ELK stack bundle being produced, here's what the test for that bundle >>> looks like: >>> https://github.com/juju-solutions/bundle-elk-stack/blob/master/tests/10-test-bundle >>> >>> This makes a lot of sense because it's asserting that the bundle is >>> working as expected by the Author who put the bundle together. It's also >>> loading the bundle.yaml as the deployment spec meaning as the bundle >>> evolves the tests will make sure they continue to run as expected. Also, >>> this could potentially be used in future smoke tests for charms being >>> updated if a CI process swaps out, say elasticsearch, for a newer version >>> of a charm being reviewed. We can assert that both the unittests in >>> elasticsearch work and it operates properly in an existing real world >>> solution a la the bundle. >>> >>> Additional examples: >>> - >>> https://github.com/juju-solutions/bundle-realtime-syslog-analytics/blob/master/tests/01-bundle.py >>> - >>> https://github.com/juju-solutions/bundle-apache-core-batch-processing/blob/master/tests/01-bundle.py >>> >>> # All charms need to have comprehensive tests (unit or amulet) >>> >>> This is just a clarification and more strongly typed policy change that >>> require charms have (preferred) unit tests or, if not applicable, then an >>> Amulet test. Bash doesn't really allow for unittesting, so in those >>> scenarios, Amulet tests would function as a valid testing case. >>> >>> There are also some charms which will not make sense as a bundle. One >>> example is the recently promulgated Fiche charm: >>> http://bazaar.launchpad.net/~charmers/charms/trusty/fiche/trunk/view/head:/tests/10-deploy >>> It's >>> a standalone pastebin, but it's an awesome service that provides deployment >>> validation with an Amulet test. The test stands up the charm, exercises >>> configuration, and validates the service responds in an expected way. For >>> scenarios where a charm does not have a bundle an Amulet test would be >>> required. >>> >>> Any charm that currently includes an Amulet test is welcome to continue >>> keeping such a test. >>> >>> # All charms should be included in a bundle >>> >>> This last one is to underscore that charms need to serve a purpose. This >>> policy is written as not an absolute, but instead a strongly worded >>> suggestion as there are always charms that are exceptions to the rules. One >>> such example is the aforementioned Fiche charm which as a bundle would not >>> make as much sense, but is still a purposeful charm. >>> >>> That being said, most users coming to consume Juju are looking to solve >>> a problem. Bundles underscore solutions to problems that people can >>> consume, and get started quicker. >>> >>> As such, when new applications are charmed a test of "is this >>> application something that serves a clear purpose" having a bundle >>> submitted alongside the charm validates that claim and provides users a way >>> to immediately get started with a solution. >>> >>> # Conclusion >>> >>> These policy changes, once accepted, will be targeted at all charms and >>> bundles in Xenial as well as any new charm submitted after policy >>> acceptance date for trusty, and finally any charm currently under review >>> will be encouraged to adhere to the new policy but won't be required. >>> >>> # Action items >>> >>> I'm seeking feedback on this concept and welcome suggestions for >>> improvements, questions, dissenting opinions, and any other remarks as well >>> as votes from ~charmers and feedback from the community at large. >>> >>> Thanks, >>> Marco Ceppi >>> >>> -- >>> Juju mailing list >>> Juju@lists.ubuntu.com >>> Modify settings or unsubscribe at: >>> https://lists.ubuntu.com/mailman/listinfo/juju >>> >>> >> >> -- >> Juju mailing list >> Juju@lists.ubuntu.com >> Modify settings or unsubscribe at: >> https://lists.ubuntu.com/mailman/listinfo/juju >> >> >
-- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
