Re: [Mesa-dev] A proposal for new testing requirements for stable releases
On 16.07.2014 06:16, Carl Worth wrote: Michel Dänzer mic...@daenzer.net writes: Quite frankly, my real concern with all of this is not that the driver maintainers will propose something bad, but that I will inadvertently botch something while cherry-picking or merging a conflict, etc. that I won't be able to notice in my touch testing. [...] But my proposal was not intended to make that a requirement. You can continue to get your commits in with no additional testing on your part, by just affirming for each release that you're OK with what the stable-branch maintainer has put together. When I phrase things that way, does it seem more reasonable to you? And if this feels like a more bureaucratic means for doing nothing effectually different than what we did before, please humor me. It does seem like that to me. (Maybe I'm just a wimp, but it's been tough at times to trust myself to resolve conflicts in code that I know I don't have any way to test. I do ask for help if the conflict looks really messy. But I don't like to bother people for things that look trivial. And even the trivial, manual conflict resolution can give me misgivings that I might be breaking the release.) I'm fine with you asking the patch author or another developer of the affected subsystem for a backport if there is any conflict, however trivial. -- Earthling Michel Dänzer| http://www.amd.com Libre software enthusiast |Mesa and X developer signature.asc Description: OpenPGP digital signature ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] A proposal for new testing requirements for stable releases
On 09.07.2014 08:10, Carl Worth wrote: 3. I'd like to receive a testing report from each driver team This is the meat of my proposal. I'm requesting that each driver team designate one (or more) people that will be responsible for running (or collecting) tests for each release-candidate and reporting the results to me. With a new release-candidate pushed by the end of the day on Monday, and me starting the actual release work on Friday, that gives 72 hours for each team to perform testing. I'm happy to let each driver team decide its own testing strategy. I would hope it would be based on running piglit across a set of interesting hardware configurations, (and augmenting piglit as necessary as bug fixes are performed). But I do not plan to be involved in the specification of what those configurations are. All I need is something along the lines of: The radeon team is OK with releasing commit foo sent to me before the scheduled day of the release. Obviously, any negative report from any team can trigger changes to the patches to be included in the release. This sounds good, but... And in order to put some teeth into this scheme: I propose that on the day of the release, the release manager drop all driver-specific patches for any driver for which the driver team has not provided an affirmative testing report. ... this seems a bit harsh. A lot of backported fixes are specific to one driver and have zero impact on anything else. Surely it should be up to the discretion of the driver maintainers whether or not such changes should be backported. OTOH, who's supposed to give that sort of OK for changes to say st/mesa or Mesa core? The former may affect all Gallium based drivers, the latter basically anything. It's unlikely that any individual or organization can test all affected setups in advance. Given those issues, and that as you say, the current process seems to have been working out pretty well overall, is introducing that kind of bureaucracy really justified? [...] who from each driver team is willing to volunteer to send me testing reports? We at AMD are working towards having such regular testing, but we're not quite there yet, nor do we have an exact schedule yet for when we will be. -- Earthling Michel Dänzer| http://www.amd.com Libre software enthusiast |Mesa and X developer ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] A proposal for new testing requirements for stable releases
On Tue, Jul 15, 2014 at 2:18 AM, Michel Dänzer mic...@daenzer.net wrote: On 09.07.2014 08:10, Carl Worth wrote: 3. I'd like to receive a testing report from each driver team This is the meat of my proposal. I'm requesting that each driver team designate one (or more) people that will be responsible for running (or collecting) tests for each release-candidate and reporting the results to me. With a new release-candidate pushed by the end of the day on Monday, and me starting the actual release work on Friday, that gives 72 hours for each team to perform testing. I'm happy to let each driver team decide its own testing strategy. I would hope it would be based on running piglit across a set of interesting hardware configurations, (and augmenting piglit as necessary as bug fixes are performed). But I do not plan to be involved in the specification of what those configurations are. All I need is something along the lines of: The radeon team is OK with releasing commit foo sent to me before the scheduled day of the release. Obviously, any negative report from any team can trigger changes to the patches to be included in the release. This sounds good, but... And in order to put some teeth into this scheme: I propose that on the day of the release, the release manager drop all driver-specific patches for any driver for which the driver team has not provided an affirmative testing report. ... this seems a bit harsh. A lot of backported fixes are specific to one driver and have zero impact on anything else. Surely it should be up to the discretion of the driver maintainers whether or not such changes should be backported. OTOH, who's supposed to give that sort of OK for changes to say st/mesa or Mesa core? The former may affect all Gallium based drivers, the latter basically anything. It's unlikely that any individual or organization can test all affected setups in advance. Perhaps that should have read s/testing//? I think all he's looking for is a yes from each driver team with changes in stable. This can be derived from your supreme confidence in your mesa-stable tagging (and Carl's super-human cherry-picking), or just seeing if the thing builds with your driver, or actually running it on any hw you deem necessary. FTR, I believe I'm the one who suggested adding teeth to this since I felt that no one would actually do any checking unless there was some reason to. The mesa/core and mesa/st changes get a free ride since (a) Carl can test them on his setup via soft/llvmpipe and (b) the shared responsibility makes it awkward. Cheers, -ilia ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/mesa-dev
Re: [Mesa-dev] A proposal for new testing requirements for stable releases
Michel Dänzer mic...@daenzer.net writes: configurations are. All I need is something along the lines of: The radeon team is OK with releasing commit foo ... [snip of the rest of my proposal] This sounds good, but... Good! Thanks for reading this and giving me some feedback. And in order to put some teeth into this scheme: I propose that on the day of the release, the release manager drop all driver-specific patches for any driver for which the driver team has not provided an affirmative testing report. ... this seems a bit harsh. A lot of backported fixes are specific to one driver and have zero impact on anything else. Surely it should be up to the discretion of the driver maintainers whether or not such changes should be backported. Yes. I do want to give the driver maintainers a lot of discretion. They are already the people who are doing the proposal of which changes should be backported. Quite frankly, my real concern with all of this is not that the driver maintainers will propose something bad, but that I will inadvertently botch something while cherry-picking or merging a conflict, etc. that I won't be able to notice in my touch testing. OTOH, who's supposed to give that sort of OK for changes to say st/mesa or Mesa core? The former may affect all Gallium based drivers, the latter basically anything. Previously, for each stable release, I was doing piglit testing against the hardware driver for my personal laptop. My hope was that that would give at least some coverage for any mesa-core changes. With the current release cycle, I've expanded that testing to also include piglit runs against swrast (in both dri and gallium flavors). I hope that that will give us sufficient confidence in non-driver-specific changes. With this additional testing, it's sufficiently time-intensive that I don't plan to repeat it in a single cycle, (I'm performing that testing before pushing out a proposed branch like I did yesterday). That's why my email yesterday said I could still accept driver-specific changes, (but any core changes will have to wait for the next release). It's unlikely that any individual or organization can test all affected setups in advance. I agree. And I won't require that. All I want is what I said above: We're OK with releasing commit foo From a reasonably representative of each driver team. I'm not going to ask for documentation to backup that statement, (no piglit reports or anything else). So every driver team can come up with their own criteria for comfort level. If you're happy to continue to just defer to me (or any other stable-release manager), then all I'm asking for is one quick email each release to give your OK. If driver teams want to do some actual testing, then this new process gives them time to do that testing and a way to communicate those results prior to release. And I think that's the most-important change here. I'm committing to providing a 3-day window in which testing can be performed prior to each stable release. We at AMD are working towards having such regular testing, but we're not quite there yet, nor do we have an exact schedule yet for when we will be. It's great that you're working toward it. I think that's a valuable goal that every driver team should work toward. But my proposal was not intended to make that a requirement. You can continue to get your commits in with no additional testing on your part, by just affirming for each release that you're OK with what the stable-branch maintainer has put together. When I phrase things that way, does it seem more reasonable to you? And if this feels like a more bureaucratic means for doing nothing effectually different than what we did before, please humor me. I think it can help reduce some psychological pressure for the release maintainer. (Maybe I'm just a wimp, but it's been tough at times to trust myself to resolve conflicts in code that I know I don't have any way to test. I do ask for help if the conflict looks really messy. But I don't like to bother people for things that look trivial. And even the trivial, manual conflict resolution can give me misgivings that I might be breaking the release.) -Carl pgpN7MeBSwt6B.pgp Description: PGP signature ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/mesa-dev
[Mesa-dev] A proposal for new testing requirements for stable releases
I've been doing stable-branch release of mesa for close to a year now. In all of that time, there's one thing that I've never been very comfortable with. Namely, release candidates are not tested very thoroughly prior to release.[*] I'm glad that we haven't had any major problems yet with broken releases. But I'd like to improve our release testing rather than just trust to future luck. Here are the changes that I'm proposing now for comments: 1. The release schedule needs to be predictable. The current ideal is every 2 weeks for stable releases. I believe that is healthy and feasible. I propose that releases occur consistently every 2 weeks on Friday, (in the release manager's timezone---currently America/Los_Angeles). 2. The release candidate needs to be made available in advance to allow for testing. I propose that the release manager push a candidate branch to the upstream repository by the end of the Monday prior to each scheduled Friday release. 3. I'd like to receive a testing report from each driver team This is the meat of my proposal. I'm requesting that each driver team designate one (or more) people that will be responsible for running (or collecting) tests for each release-candidate and reporting the results to me. With a new release-candidate pushed by the end of the day on Monday, and me starting the actual release work on Friday, that gives 72 hours for each team to perform testing. I'm happy to let each driver team decide its own testing strategy. I would hope it would be based on running piglit across a set of interesting hardware configurations, (and augmenting piglit as necessary as bug fixes are performed). But I do not plan to be involved in the specification of what those configurations are. All I need is something along the lines of: The radeon team is OK with releasing commit foo sent to me before the scheduled day of the release. Obviously, any negative report from any team can trigger changes to the patches to be included in the release. And in order to put some teeth into this scheme: I propose that on the day of the release, the release manager drop all driver-specific patches for any driver for which the driver team has not provided an affirmative testing report. Does the above sound like a reasonable scheme? If so,who from each driver team is willing to volunteer to send me testing reports? I'll be happy to send personal reminders to such volunteers as releases are getting close and testing reports are missing, (before yanking out any driver-specific patches). Thanks in advance for any thoughts or comments, -Carl PS. This same testing scheme can obviously be used for major releases as well, presumably with the same volunteers, (but different details on release timing). [*] For background, here's the testing scheme I've been using so far: 1. Running each release candidate through piglit on my laptop 2. Trying to leave the stable branch pushed out for at least a day or so to allow testing prior to release. The testing I do on my laptop at least gives some sanity checking for core, but very little driver-specific testing. Meanwhile, the majority of stable-branch fixes seem to be in driver-specific code. And I don't know if there is any serious testing of the stable branch by anyone else between when I push it and when I release, (I haven't heard much either way). I know it doesn't help that the timing of my releases has not often been predictable. ___ mesa-dev mailing list mesa-dev@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/mesa-dev
