Re: [openstack-dev] [stestr?][tox?][infra?] Unexpected success isn't a failure

2018-08-03 Thread Matthew Treinish
On Tue, Jul 10, 2018 at 03:16:14PM -0400, Matthew Treinish wrote: > On Tue, Jul 10, 2018 at 10:16:37AM +0100, Chris Dent wrote: > > On Mon, 9 Jul 2018, Matthew Treinish wrote: > > > > > It's definitely a bug, and likely a bug in stestr (or one of the lower > > > level > > > packages like testtoo

Re: [openstack-dev] [stestr?][tox?][infra?] Unexpected success isn't a failure

2018-07-10 Thread Matthew Treinish
On Tue, Jul 10, 2018 at 10:16:37AM +0100, Chris Dent wrote: > On Mon, 9 Jul 2018, Matthew Treinish wrote: > > > It's definitely a bug, and likely a bug in stestr (or one of the lower > > level > > packages like testtools or python-subunit), because that's what's generating > > the return code. T

Re: [openstack-dev] [stestr?][tox?][infra?] Unexpected success isn't a failure

2018-07-10 Thread Chris Dent
On Mon, 9 Jul 2018, Matthew Treinish wrote: It's definitely a bug, and likely a bug in stestr (or one of the lower level packages like testtools or python-subunit), because that's what's generating the return code. Tox just looks at the return code from the commands to figure out if things were

Re: [openstack-dev] [stestr?][tox?][infra?] Unexpected success isn't a failure

2018-07-09 Thread Matthew Treinish
On Mon, Jul 09, 2018 at 06:59:42PM -0500, Eric Fried wrote: > In gabbi, there's a way [1] to mark a test as an expected failure, which > makes it show up in your stestr run thusly: > > {0} > nova.tests.functional.api.openstack.placement.test_placement_api.allocations-1.28_put_that_allocation_to_ne

[openstack-dev] [stestr?][tox?][infra?] Unexpected success isn't a failure

2018-07-09 Thread Eric Fried
In gabbi, there's a way [1] to mark a test as an expected failure, which makes it show up in your stestr run thusly: {0} nova.tests.functional.api.openstack.placement.test_placement_api.allocations-1.28_put_that_allocation_to_new_consumer.test_request [0.710821s] ... ok == Totals == Ran: