On Thu, Feb 13, 2014 at 9:59 AM, Walter A. Boring IV <walter.bor...@hp.com> wrote: > On 02/13/2014 02:51 AM, Thierry Carrez wrote: >> >> John Griffith wrote: >>> >>> So we've talked about this a bit and had a number of ideas regarding >>> how to test and show compatibility for third-party drivers in Cinder. >>> This has been an eye opening experience (the number of folks that have >>> NEVER run tempest before, as well as the problems uncovered now that >>> they're trying it). >>> >>> I'm even more convinced now that having vendors run these tests is a >>> good thing and should be required. That being said there's a ton of >>> push back from my proposal to require that results from a successful >>> run of the tempest tests to accompany any new drivers submitted to >>> Cinder. >> >> Could you describe the nature of the pushback ? Is it that the tests are >> too deep and reject valid drivers ? Is it that it's deemed unfair to >> block new drivers while the existing ones aren't better ? Is it that >> it's difficult for them to run those tests and get a report ? Or is it >> because they care more about having their name covered in mainline and >> not so much about having the code working properly ? >> >>> The consensus from the Cinder community for now is that we'll >>> log a bug for each driver after I3, stating that it hasn't passed >>> certification tests. We'll then have a public record showing >>> drivers/vendors that haven't demonstrated functional compatibility, >>> and in order to close those bugs they'll be required to run the tests >>> and submit the results to the bug in Launchpad. >>> >>> So, this seems to be the approach we're taking for Icehouse at least, >>> it's far from ideal IMO, however I think it's still progress and it's >>> definitely exposed some issues with how drivers are currently >>> submitted to Cinder so those are positive things that we can learn >>> from and improve upon in future releases. >>> >>> To add some controversy and keep the original intent of having only >>> known tested and working drivers in the Cinder release, I am going to >>> propose that any driver that has not submitted successful functional >>> testing by RC1 that that driver be removed. I'd at least like to see >>> driver maintainers try... if the test fails a test or two that's >>> something that can be discussed, but it seems that until now most >>> drivers just flat out are not even being tested. >> >> I think there are multiple stages here. >> >> Stage 0: noone knows if drivers work >> Stage 1: we know the (potentially sad) state of the drivers that are in >> the release >> Stage 2: only drivers that pass tests are added, drivers that don't pass >> tests have a gap analysis and a plan to fix it >> Stage 3: drivers that fail tests are removed before release >> Stage 4: 3rd-party testing rigs must run tests on every change in order >> to stay in tree >> >> At the very minimum you should be at stage 1 for the Icehouse release, >> so I agree with your last paragraph. I'd recommend that you start the >> Juno cycle at stage 2 (for new drivers), and try to reach stage 3 for >> the end of the Juno release. >> > I have to agree with Thierry here. I think if we can get drivers to pass > the tests > in the Juno timeframe, then it's fine to remove then during Juno. > I think the idea of having drivers run their code through tempest and work > towards passing all of those tests is a great thing for Cinder and OpenStack > in general. > > What I would do different for the Icehouse release is this: > > If a driver doesn't pass the certification test by IceHouse RC1, then we > have a bug filed > against the driver. I would also put a warning message in the log for that > driver that it > doesn't pass the certification test. I would not remove it from the > codebase. > > Also: > if a driver hasn't even run the certification test by RC1, then we mark > the driver as > uncertified and deprecated in the code and throw an error at driver init > time. > We can have a option in cinder.conf that says > ignore_uncertified_drivers=False. > If an admin wants to ignore the error, they set the flag to True, and we let > the driver init at next startup. > The admin then takes full responsibility for running uncertified code. > > I think removing the drivers outright is premature for Icehouse, since the > certification process is a new thing. > For Juno, we remove any drivers that are still marked as uncertified and > haven't run the tests. > > I think the purpose of the tests is to get vendors to actually run their > code through tempest and > prove to the community that they are willing to show that they are fixing > their code. At the end of the day, > it better serves the community and Cinder if we have many working drivers. > > My $0.02, > Walt > > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev I'm fine with all of the recommendations above, however I do want to point out that having your driver/device work in OpenStack should not be something new to you. That's what's so disappointing IMO. And while a cert script and a process is *newish* we talked about this at the summit in Portland almost a year ago. Also, Tempest and the gating infrastructure that we have in place is not new either.
IMO it's poor community behavior to: 1. Not test your code 2. Not want to fix your code All of that being said, I'll stop my rant at this point. I'm absolutely fine with the proposals here. It's just disappointing that people are willing to submit and release code without testing and even worse knowing that it doesn't work. Thanks, John _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev