Re: [Openstack-qa-team] Policy for Tests linked to existing Bugs
On 5/7/2012 2:29 PM, Venkatesan, Ravikumar wrote: On 5/4/2012 9:08 AM, Jay Pipes wrote: On 05/04/2012 06:14 AM, Karajgi, Rohit wrote: Hi, What is the policy that we should or are following for test cases that fail due to an existing Open Bug in Launchpad? For eg: https://github.com/openstack/tempest/blob/master/tempest/tests/test_list_floating_ips.py#L64 skips the test and posts the Bug ID in the message. Do we submit the test for review with the @skip(until Bug #xyz is fixed) decorator applied? This is the choice that I believe is the easiest and simplest. We just need to be vigilant to remove the skips when/if the bug is fixed. -jay Jay, I agree with the idea that we should check the test in anyway. But we have the issue of who remembers to remove the skip decorator when the bug is fixed. Since we are going to be running tempest nightly, I think it might be better to make these tests fail in the absence of the bug rather than being skipped. I think there may already be some cases in Tempest that do this. -David I agree with David, and I prefer that the test failed rather than using skip decorator. - Ravi Ravi, after hearing Jays comments I realized that we can't do this if we are going to use tempest to gate trunk checkins for other projects. If we make the test fail, then it will be impossible for a developer to check in a fix to a bug unless they also change the tempest test. In an ideal world this would be the case, as surely they already do this for unit tests, but I don't think we are ready for that. I thought I sent a message to that effect last week but perhaps it never went through. -David -- Mailing list: https://launchpad.net/~openstack-qa-team Post to : openstack-qa-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack-qa-team More help : https://help.launchpad.net/ListHelp
Re: [Openstack-qa-team] Policy for Tests linked to existing Bugs
On 5/4/2012 9:08 AM, Jay Pipes wrote: > On 05/04/2012 06:14 AM, Karajgi, Rohit wrote: >> Hi, >> >> What is the policy that we should or are following for test cases that >> fail due to an existing Open Bug in Launchpad? >> >> For eg: >> https://github.com/openstack/tempest/blob/master/tempest/tests/test_list_floating_ips.py#L64 >> >> >> skips the test and posts the Bug ID in the message. >> >> Do we submit the test for review with the @skip(until Bug #xyz is fixed) >> decorator applied? > > This is the choice that I believe is the easiest and simplest. We just > need to be vigilant to remove the skips when/if the bug is fixed. > > -jay > > Jay, I agree with the idea that we should check the test in anyway. But we have the issue of who remembers to remove the skip decorator when the bug is fixed. Since we are going to be running tempest nightly, I think it might be better to make these tests fail in the absence of the bug rather than being skipped. I think there may already be some cases in Tempest that do this. -David I agree with David, and I prefer that the test failed rather than using skip decorator. - Ravi -- Mailing list: https://launchpad.net/~openstack-qa-team Post to : openstack-qa-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack-qa-team More help : https://help.launchpad.net/ListHelp -- Mailing list: https://launchpad.net/~openstack-qa-team Post to : openstack-qa-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack-qa-team More help : https://help.launchpad.net/ListHelp
Re: [Openstack-qa-team] Policy for Tests linked to existing Bugs
Yes, I suppose it depends on how development teams think about tempest. In an ideal world, with complete test coverage of everything, any commit that fixes a bug would also have to unskip or change some test. But we can't expect developers to run tempest before a commit in the same way they would run unit tests so I withdraw my suggestion. We can just skip the tests and be watchful. I am not sure if it would be worth the trouble, but we could have something that watches bug tickets for "Fix Released" and notifies tempest maintainers if that bug is being skipped. -David On 5/4/2012 9:38 AM, Jay Pipes wrote: On 05/04/2012 09:28 AM, David Kranz wrote: On 5/4/2012 9:08 AM, Jay Pipes wrote: On 05/04/2012 06:14 AM, Karajgi, Rohit wrote: Hi, What is the policy that we should or are following for test cases that fail due to an existing Open Bug in Launchpad? For eg: https://github.com/openstack/tempest/blob/master/tempest/tests/test_list_floating_ips.py#L64 skips the test and posts the Bug ID in the message. Do we submit the test for review with the @skip(until Bug #xyz is fixed) decorator applied? This is the choice that I believe is the easiest and simplest. We just need to be vigilant to remove the skips when/if the bug is fixed. -jay Jay, I agree with the idea that we should check the test in anyway. But we have the issue of who remembers to remove the skip decorator when the bug is fixed. Since we are going to be running tempest nightly, I think it might be better to make these tests fail in the absence of the bug rather than being skipped. I think there may already be some cases in Tempest that do this. Well, we're going to be running Tempest nightly, but also a subset for every commit into any of the core projects... and that subset will be a full merge gate. So, a further question to answer is whether we treat *some* of the tests differently than others with respect to how skips are handled? Best, -jay -- Mailing list: https://launchpad.net/~openstack-qa-team Post to : openstack-qa-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack-qa-team More help : https://help.launchpad.net/ListHelp
Re: [Openstack-qa-team] Policy for Tests linked to existing Bugs
On 05/04/2012 09:28 AM, David Kranz wrote: On 5/4/2012 9:08 AM, Jay Pipes wrote: On 05/04/2012 06:14 AM, Karajgi, Rohit wrote: Hi, What is the policy that we should or are following for test cases that fail due to an existing Open Bug in Launchpad? For eg: https://github.com/openstack/tempest/blob/master/tempest/tests/test_list_floating_ips.py#L64 skips the test and posts the Bug ID in the message. Do we submit the test for review with the @skip(until Bug #xyz is fixed) decorator applied? This is the choice that I believe is the easiest and simplest. We just need to be vigilant to remove the skips when/if the bug is fixed. -jay Jay, I agree with the idea that we should check the test in anyway. But we have the issue of who remembers to remove the skip decorator when the bug is fixed. Since we are going to be running tempest nightly, I think it might be better to make these tests fail in the absence of the bug rather than being skipped. I think there may already be some cases in Tempest that do this. Well, we're going to be running Tempest nightly, but also a subset for every commit into any of the core projects... and that subset will be a full merge gate. So, a further question to answer is whether we treat *some* of the tests differently than others with respect to how skips are handled? Best, -jay -- Mailing list: https://launchpad.net/~openstack-qa-team Post to : openstack-qa-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack-qa-team More help : https://help.launchpad.net/ListHelp
Re: [Openstack-qa-team] Policy for Tests linked to existing Bugs
On 5/4/2012 9:08 AM, Jay Pipes wrote: On 05/04/2012 06:14 AM, Karajgi, Rohit wrote: Hi, What is the policy that we should or are following for test cases that fail due to an existing Open Bug in Launchpad? For eg: https://github.com/openstack/tempest/blob/master/tempest/tests/test_list_floating_ips.py#L64 skips the test and posts the Bug ID in the message. Do we submit the test for review with the @skip(until Bug #xyz is fixed) decorator applied? This is the choice that I believe is the easiest and simplest. We just need to be vigilant to remove the skips when/if the bug is fixed. -jay Jay, I agree with the idea that we should check the test in anyway. But we have the issue of who remembers to remove the skip decorator when the bug is fixed. Since we are going to be running tempest nightly, I think it might be better to make these tests fail in the absence of the bug rather than being skipped. I think there may already be some cases in Tempest that do this. -David -- Mailing list: https://launchpad.net/~openstack-qa-team Post to : openstack-qa-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack-qa-team More help : https://help.launchpad.net/ListHelp
Re: [Openstack-qa-team] Policy for Tests linked to existing Bugs
On 05/04/2012 06:14 AM, Karajgi, Rohit wrote: Hi, What is the policy that we should or are following for test cases that fail due to an existing Open Bug in Launchpad? For eg: https://github.com/openstack/tempest/blob/master/tempest/tests/test_list_floating_ips.py#L64 skips the test and posts the Bug ID in the message. Do we submit the test for review with the @skip(until Bug #xyz is fixed) decorator applied? This is the choice that I believe is the easiest and simplest. We just need to be vigilant to remove the skips when/if the bug is fixed. -jay -- Mailing list: https://launchpad.net/~openstack-qa-team Post to : openstack-qa-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack-qa-team More help : https://help.launchpad.net/ListHelp
