DW> I've been reviewing the test failures from the gating job and what DW> concerns me is that at least some of them are legitimate.
Yeah, indeed, I didn't mean to minimize the severity of the actual issues, I just wonder if gating new tempest changes on whether or not the suite runs "right now" makes total sense. For unit tests, it does, but since tempest and nova are decoupled, it's a bit of a strange relationship :) In other suites, I've seen an XFAIL result used to mark tests that we know are failing right now so that they're not SKIPped like tests that are missing some component, but rather just not fatal to the task at hand. Maybe something like that would be useful in tempest? If I found a bug in Nova right now and wanted to get a test into tempest ASAP to poke it, submitting as XFAIL would (a) not break Jenkins because the test failed (as expected) and (b) raise a flag when the test started to pass to make sure that it gets un-marked as XFAIL. DW> They may not be caused by the patch at hand, but servers and volumes DW> going into error status definitely signal issues, whether they be in DW> code or environment. I don't have access to the Tempest CI DW> environment so I don't have much insight into those issues DW> specifically, though there might be some additional error checking DW> that we can do to get more information on what is going wrong. Yeah, I've been trying to reproduce the issues locally, as I'm happy to fix them up if I can figure out what the problem is. However, I feel like I'm flying blind a bit, without a view into the CI machine itself :) DW> I'm doing what I can Dan to get your patches reviewed. The trick DW> being that since there is a dependency chain between most of the DW> commits, it adds a level of complexity. Jay, who's done most of the DW> CI setup thus far, is out of country, so I'm trying to find other DW> folks I can reach out to help stabilize the environment. Yeah, where is that slacker? :) We're very appreciative of the review you've already provided, as well as your efforts to help move those forward. I didn't intend to come off whiny, I just wanted to make sure that we don't get too far down the rabbit hole, and certainly don't want to get ignored because of the big red "-1" next to us :) David Kranz and I have been chatting on IRC today about hopefully marking the errant tests as skip so that we can move forward. Thanks! -- Dan Smith IBM Linux Technology Center -- Mailing list: https://launchpad.net/~openstack-qa-team Post to : openstack-qa-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack-qa-team More help : https://help.launchpad.net/ListHelp