Re: Makefile target names
On Thu, Jan 22, 2015 at 04:57:36PM +, Marco Ceppi wrote: > test: lint unit-test functional-test -1, I'd rather 'test' be unit testing only. Many charms have this already and it seems like unecessary busy work to change it. > ``` > makefile: > - code-lint > - unit-test > ``` -1, vote for 'lint', 'test' (unit test only) at this level, and agree with Tim that it's redundnat to print these out, we should make the recommeneded defaults what bundletester supports. All else I agree with. -- David Britton -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Re: Makefile target names
Marco, I like your proposal with one change - we don't need the test.yaml changes. Instead I would suggest we add 'unit-test' to the list of default bundletester targets. So bundletester will run proof, lint, test, and unit-test (charm author should choose test or unit-test, not both). Bundletester will *not* run functional-test, since it runs everything +x in the tests/ dir, which almost always overlaps with functional-test. On Thu, Jan 22, 2015 at 11:57 AM, Marco Ceppi wrote: > We can also add Makefile checking to charm proof, for an even greater > redundancy. > > To avoid multiple invocations of charm proof (not terrible, IMO) lint > could be broken down further: > > lint: proof code_lint > > proof: > charm proof > > code_lint: > # Your code here > > Then have bundle tester sniff out code_lint, or use the test.yaml > configuration to point lint to code_lint. Doesn't change UX for the > author/contributor but does add a level of complexity. It seems like > Makefile's are the overwhelming method for consolidating tasks for charms, > I'd like to kick off the following proposal for Makefile format to be > placed in charm create templates: > > ``` > test: lint unit-test functional-test > lint: proof code-lint > sync: charm-helpers-sync > > code-lint: > # FILL IN COMMANDS FOR PERFORMING CODE LINT > > unit-test: ># COMMANDS REQUIRED TO UNIT TEST > > charm-helpers-sync: > @scripts/sync.py > > functional-test: > juju test > > proof: > charm proof > ``` > > With a test.yml file that contained the following: > > ``` > makefile: > - code-lint > - unit-test > ``` > > And where applicable, add a .venv target for python charms and recommend > the use of having charm deps modeled in requirements.txt and pip installed > to that virtualenv. > > Opinions, additions, concerns? > > On Thu Jan 22 2015 at 11:41:56 AM Wes Mason > wrote: > >> On 22 January 2015 at 16:36, Simon Davy wrote: >> >>> On 22 January 2015 at 16:29, David Britton >>> wrote: >>> > On Thu, Jan 22, 2015 at 04:17:26PM +, Simon Davy wrote: >>> >> On 22 January 2015 at 15:13, David Britton < >>> david.brit...@canonical.com> wrote: >>> >> > >>> >> > lint: >>> >> > - make lint >>> >> > >>> >> >>> >> Could we also make[1] the charm linter lint the makefile for the >>> >> presence of targets agreed in the outcome of this thread? >>> > >>> > "charm proof" >>> > >>> > I like it. (bundle tester already runs this) >>> >>> Which is interesting, as my lint targets general runs charm proof too, >>> so it'd be run twice in that case? >>> >>> Not a big issue, but if the charm store/review queue is automatically >>> charm-proofing too, perhaps the make lint target should not be? >>> >>> -- >>> Simon >>> >>> >> Whelp it's still nice to have as part of lint when developing the charm, >> and charm-proof isn't exactly the slowest process to run multiple times. >> >> -- >> Juju mailing list >> Juju@lists.ubuntu.com >> Modify settings or unsubscribe at: https://lists.ubuntu.com/ >> mailman/listinfo/juju >> > > -- > Juju mailing list > Juju@lists.ubuntu.com > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/juju > > -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Re: Makefile target names
We can also add Makefile checking to charm proof, for an even greater redundancy. To avoid multiple invocations of charm proof (not terrible, IMO) lint could be broken down further: lint: proof code_lint proof: charm proof code_lint: # Your code here Then have bundle tester sniff out code_lint, or use the test.yaml configuration to point lint to code_lint. Doesn't change UX for the author/contributor but does add a level of complexity. It seems like Makefile's are the overwhelming method for consolidating tasks for charms, I'd like to kick off the following proposal for Makefile format to be placed in charm create templates: ``` test: lint unit-test functional-test lint: proof code-lint sync: charm-helpers-sync code-lint: # FILL IN COMMANDS FOR PERFORMING CODE LINT unit-test: # COMMANDS REQUIRED TO UNIT TEST charm-helpers-sync: @scripts/sync.py functional-test: juju test proof: charm proof ``` With a test.yml file that contained the following: ``` makefile: - code-lint - unit-test ``` And where applicable, add a .venv target for python charms and recommend the use of having charm deps modeled in requirements.txt and pip installed to that virtualenv. Opinions, additions, concerns? On Thu Jan 22 2015 at 11:41:56 AM Wes Mason wrote: > On 22 January 2015 at 16:36, Simon Davy wrote: > >> On 22 January 2015 at 16:29, David Britton >> wrote: >> > On Thu, Jan 22, 2015 at 04:17:26PM +, Simon Davy wrote: >> >> On 22 January 2015 at 15:13, David Britton < >> david.brit...@canonical.com> wrote: >> >> > >> >> > lint: >> >> > - make lint >> >> > >> >> >> >> Could we also make[1] the charm linter lint the makefile for the >> >> presence of targets agreed in the outcome of this thread? >> > >> > "charm proof" >> > >> > I like it. (bundle tester already runs this) >> >> Which is interesting, as my lint targets general runs charm proof too, >> so it'd be run twice in that case? >> >> Not a big issue, but if the charm store/review queue is automatically >> charm-proofing too, perhaps the make lint target should not be? >> >> -- >> Simon >> >> > Whelp it's still nice to have as part of lint when developing the charm, > and charm-proof isn't exactly the slowest process to run multiple times. > > -- > Juju mailing list > Juju@lists.ubuntu.com > Modify settings or unsubscribe at: https://lists.ubuntu.com/ > mailman/listinfo/juju > -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Re: Makefile target names
On 22 January 2015 at 16:36, Simon Davy wrote: > On 22 January 2015 at 16:29, David Britton > wrote: > > On Thu, Jan 22, 2015 at 04:17:26PM +, Simon Davy wrote: > >> On 22 January 2015 at 15:13, David Britton > wrote: > >> > > >> > lint: > >> > - make lint > >> > > >> > >> Could we also make[1] the charm linter lint the makefile for the > >> presence of targets agreed in the outcome of this thread? > > > > "charm proof" > > > > I like it. (bundle tester already runs this) > > Which is interesting, as my lint targets general runs charm proof too, > so it'd be run twice in that case? > > Not a big issue, but if the charm store/review queue is automatically > charm-proofing too, perhaps the make lint target should not be? > > -- > Simon > > Whelp it's still nice to have as part of lint when developing the charm, and charm-proof isn't exactly the slowest process to run multiple times. -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Re: Makefile target names
Same here, the OpenStack charms have charm proof in the lint target. I expect it would be run twice in that case. On Thu, Jan 22, 2015 at 10:36 AM, Simon Davy wrote: > On 22 January 2015 at 16:29, David Britton > wrote: > > On Thu, Jan 22, 2015 at 04:17:26PM +, Simon Davy wrote: > >> On 22 January 2015 at 15:13, David Britton > wrote: > >> > > >> > lint: > >> > - make lint > >> > > >> > >> Could we also make[1] the charm linter lint the makefile for the > >> presence of targets agreed in the outcome of this thread? > > > > "charm proof" > > > > I like it. (bundle tester already runs this) > > Which is interesting, as my lint targets general runs charm proof too, > so it'd be run twice in that case? > > Not a big issue, but if the charm store/review queue is automatically > charm-proofing too, perhaps the make lint target should not be? > > -- > Simon > -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Re: Makefile target names
On 22 January 2015 at 16:29, David Britton wrote: > On Thu, Jan 22, 2015 at 04:17:26PM +, Simon Davy wrote: >> On 22 January 2015 at 15:13, David Britton >> wrote: >> > >> > lint: >> > - make lint >> > >> >> Could we also make[1] the charm linter lint the makefile for the >> presence of targets agreed in the outcome of this thread? > > "charm proof" > > I like it. (bundle tester already runs this) Which is interesting, as my lint targets general runs charm proof too, so it'd be run twice in that case? Not a big issue, but if the charm store/review queue is automatically charm-proofing too, perhaps the make lint target should not be? -- Simon -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Re: Makefile target names
On Thu, Jan 22, 2015 at 04:17:26PM +, Simon Davy wrote: > On 22 January 2015 at 15:13, David Britton > wrote: > > > > lint: > > - make lint > > > > Could we also make[1] the charm linter lint the makefile for the > presence of targets agreed in the outcome of this thread? "charm proof" I like it. (bundle tester already runs this) -- David Britton -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Re: Makefile target names
Yep absolutely, we want to make sure we're in line with the existing tools. After talking with Tim, the way I understand it, bundletester doesn't use the Makefile for amulet tests. Instead, it execs everything which is +x in the tests dir (a la juju test). On Thu, Jan 22, 2015 at 9:51 AM, Charles Butler < charles.but...@canonical.com> wrote: > > On Thu, Jan 22, 2015 at 10:13 AM, David Britton < > david.brit...@canonical.com> wrote: > >> functional tests: >> - make functional-test >> > > We need to be careful about things like this - as bundletester is already > looking in tests/ for the amulet suite and might end up running the > integration tests multiple times according to the make target + > bundletesters sniffer. This is kind of an overlap of concerns with how > bundletester works and I feel warrants caution. > > Tim would know best on the behavior here, and can recommend / amend as > required. > > > > > -- > All the best, > > Charles Butler - Juju Charmer > Come see the future of datacenter orchestration: http://jujucharms.com > -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Re: Makefile target names
On 22 January 2015 at 15:13, David Britton wrote: > > lint: > - make lint > Could we also make[1] the charm linter lint the makefile for the presence of targets agreed in the outcome of this thread? [1] Pun fully intended :) -- Simon -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Re: Makefile target names
On Thu, Jan 22, 2015 at 10:13 AM, David Britton wrote: > functional tests: > - make functional-test > We need to be careful about things like this - as bundletester is already looking in tests/ for the amulet suite and might end up running the integration tests multiple times according to the make target + bundletesters sniffer. This is kind of an overlap of concerns with how bundletester works and I feel warrants caution. Tim would know best on the behavior here, and can recommend / amend as required. -- All the best, Charles Butler - Juju Charmer Come see the future of datacenter orchestration: http://jujucharms.com -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Re: Makefile target names
+1, but I would propose using hyphens for word separators, not underscores -- at least for the recommendation. I would also recommend *not* having multiple default names. As mentioned, the yaml control file I think can be used to override all this, so it still leaves room for individual preferences on the exact namings. unit tests: - make test unit tests dependencies: - make test-depends functional tests: - make functional-test lint: - make lint charm-helpers upstream sync: - make sync -- David Britton -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Re: Makefile target names
On 01/22/2015 03:57 PM, Ryan Beisner wrote: > Thanks for pointing out the yaml control file, that could be useful. But > before we make any modifications to the OpenStack charms, I think it would > be helpful to have an agreed-upon convention for the following in terms of > Makefile target names: > >- nose / unit tests > - make test > - make unit_test > - Both are in use. > - 2 cents: I would reserve both of these for unit tests, never for > amulet tests. +1. >- lint checks > - make lint > - Already unified on this afaict. +1 >- amulet tests >- make test > - make functional_test > - Both are in use. > - 2 cents: I think functional_test leaves no question as to usage. >- charm-helpers sync +1 for functional_test Also, at the recent mtg it was mentioned that automated test suite doesn't care for 'functional_test' target, presumably because amulet tests are run with 'juju test', so make target is actually not needed. But to avoid confusion I'm +1-ing again 'functional_test. Maybe worth mentioning - the MongoDB charm I inherited has .venv target that prepares venv for unit_tests to be run. Shall this also be a standard? The issue with this approach is that only 'running' series are tested - if I'm on precise, then I'll be unit_testing precise, if I upgraded to utopic, then I'd be testing utopic, etc... In that light - shall .venv remain as it is, or? Mario -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Re: Makefile target names
Thanks for pointing out the yaml control file, that could be useful. But before we make any modifications to the OpenStack charms, I think it would be helpful to have an agreed-upon convention for the following in terms of Makefile target names: - nose / unit tests - make test - make unit_test - Both are in use. - 2 cents: I would reserve both of these for unit tests, never for amulet tests. - lint checks - make lint - Already unified on this afaict. - amulet tests - make test - make functional_test - Both are in use. - 2 cents: I think functional_test leaves no question as to usage. - charm-helpers sync - make sync - Already unified on this afaict. If there is not a documented convention, can we have the necessary discussions here to create one? Thanks again, Ryan On Wed, Jan 21, 2015 at 11:40 AM, Benjamin Saller < benjamin.sal...@canonical.com> wrote: > While convention is great there is an additional path, you can if your > project differs from the de facto standards, include an explicit list of > targets in your tests/tests.yaml file > > makefile: > - lint > - unit_test > - something_else > > That file allows customization of much of bundletesters policy. > > -Ben > > On Wed, Jan 21, 2015 at 9:05 AM, Ryan Beisner > wrote: > >> Greetings, >> >> I'd like to invite discussion on Makefile target names. I've seen a few >> different takes on Makefile target naming conventions across charms. For >> example, in the OpenStack charms, `make test` runs amulet and `make >> unit_test` performs nose tests. In many/most other charms, `make test` >> infers unit/nose testing, and amulet target names can vary. >> >> As I understand bundletester: it expects `make test` to be unit tests. >> Amulet targets in the Makefile aren't processed if they exist. Instead, >> the executables in the test dir are fired off. And, I think that should >> all be quite fine as long as the charm's amulet make target isn't doing >> anything important. >> >> The net effect for OpenStack charms at the moment is that when they hit >> Juju QA, amulet fires off twice, and unit is not run. I'd like to make >> sure the OpenStack charms are in line with any established Makefile >> convention. Is there a reference or doc for such a convention? >> >> I've seen 'unit_test' and 'functional_test' target names in use as well, >> and I quite like those, as they leave no question as to purpose. >> >> To work around the variations we've seen across charms, server team's >> OSCI (OpenStack CI charm testing) ignores make target names, and instead >> parses the Makefile, looking for the right "thing-to-do," then execs the >> target found. Bear in mind that OSCI isn't intended to be a replacement >> for general charm QA, rather it is an intense safety trigger for the >> OpenStack charm developers. We also want these charms to succeed in Juju >> QA / CI. >> >> Input and advice are much appreciated! >> >> Many thanks, >> >> >> Ryan Beisner >> >> >> -- >> Juju mailing list >> Juju@lists.ubuntu.com >> Modify settings or unsubscribe at: >> https://lists.ubuntu.com/mailman/listinfo/juju >> >> > -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Re: Makefile target names
While convention is great there is an additional path, you can if your project differs from the de facto standards, include an explicit list of targets in your tests/tests.yaml file makefile: - lint - unit_test - something_else That file allows customization of much of bundletesters policy. -Ben On Wed, Jan 21, 2015 at 9:05 AM, Ryan Beisner wrote: > Greetings, > > I'd like to invite discussion on Makefile target names. I've seen a few > different takes on Makefile target naming conventions across charms. For > example, in the OpenStack charms, `make test` runs amulet and `make > unit_test` performs nose tests. In many/most other charms, `make test` > infers unit/nose testing, and amulet target names can vary. > > As I understand bundletester: it expects `make test` to be unit tests. > Amulet targets in the Makefile aren't processed if they exist. Instead, > the executables in the test dir are fired off. And, I think that should > all be quite fine as long as the charm's amulet make target isn't doing > anything important. > > The net effect for OpenStack charms at the moment is that when they hit > Juju QA, amulet fires off twice, and unit is not run. I'd like to make > sure the OpenStack charms are in line with any established Makefile > convention. Is there a reference or doc for such a convention? > > I've seen 'unit_test' and 'functional_test' target names in use as well, > and I quite like those, as they leave no question as to purpose. > > To work around the variations we've seen across charms, server team's OSCI > (OpenStack CI charm testing) ignores make target names, and instead parses > the Makefile, looking for the right "thing-to-do," then execs the target > found. Bear in mind that OSCI isn't intended to be a replacement for > general charm QA, rather it is an intense safety trigger for the OpenStack > charm developers. We also want these charms to succeed in Juju QA / CI. > > Input and advice are much appreciated! > > Many thanks, > > > Ryan Beisner > > > -- > Juju mailing list > Juju@lists.ubuntu.com > Modify settings or unsubscribe at: > https://lists.ubuntu.com/mailman/listinfo/juju > > -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Makefile target names
Greetings, I'd like to invite discussion on Makefile target names. I've seen a few different takes on Makefile target naming conventions across charms. For example, in the OpenStack charms, `make test` runs amulet and `make unit_test` performs nose tests. In many/most other charms, `make test` infers unit/nose testing, and amulet target names can vary. As I understand bundletester: it expects `make test` to be unit tests. Amulet targets in the Makefile aren't processed if they exist. Instead, the executables in the test dir are fired off. And, I think that should all be quite fine as long as the charm's amulet make target isn't doing anything important. The net effect for OpenStack charms at the moment is that when they hit Juju QA, amulet fires off twice, and unit is not run. I'd like to make sure the OpenStack charms are in line with any established Makefile convention. Is there a reference or doc for such a convention? I've seen 'unit_test' and 'functional_test' target names in use as well, and I quite like those, as they leave no question as to purpose. To work around the variations we've seen across charms, server team's OSCI (OpenStack CI charm testing) ignores make target names, and instead parses the Makefile, looking for the right "thing-to-do," then execs the target found. Bear in mind that OSCI isn't intended to be a replacement for general charm QA, rather it is an intense safety trigger for the OpenStack charm developers. We also want these charms to succeed in Juju QA / CI. Input and advice are much appreciated! Many thanks, Ryan Beisner -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju