Re: Criterion proposal: security
On Mon, 2012-11-12 at 04:10 -0500, Kamil Paral wrote: As this got generally ack'ed and no-one complained, I've pushed it into production now in the Final criteria - https://fedoraproject.org/wiki/Fedora_18_Final_Release_Criteria . I added a hyperlink to the severity classification document. D'oh, thanks for catching that. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: Criterion proposal: security
As this got generally ack'ed and no-one complained, I've pushed it into production now in the Final criteria - https://fedoraproject.org/wiki/Fedora_18_Final_Release_Criteria . I added a hyperlink to the severity classification document. -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: Criterion proposal: security
On Fri, 2012-10-26 at 12:49 -0700, Adam Williamson wrote: On Fri, 2012-10-26 at 12:44 -0700, Adam Williamson wrote: I think with the feedback we've seen so far that we can say the original proposal was substantially too broad, so how about this as a revised proposal - for now, we just add a single Final release criterion which reads: The release must contain no known security issues of 'important' or higher impact according to the Red Hat severity classification scale which cannot be satisfactorily resolved by a package update (e.g. issues during installation) ? How does that sound to everyone? It drops the issue entirely for Alpha and Beta, and means we only consider bad issues that cannot be fixed with an update for Final. Hmm, actually, let's change 'issues' to 'bugs' there, I think that makes it clearer that we're talking about things that have actually been accepted as bugs - it avoids any suggestion we'd be wading into the debate about what actually constitutes a security issue, as Johann was concerned about. So: The release must contain no known security bugs of 'important' or higher impact according to the Red Hat severity classification scale which cannot be satisfactorily resolved by a package update (e.g. issues during installation) with the understanding that QA would never use this to wade into something like the sshd question and declare that it was a Bug That Must Be Fixed. It applies only to things that are clearly agreed to be actual bugs. As this got generally ack'ed and no-one complained, I've pushed it into production now in the Final criteria - https://fedoraproject.org/wiki/Fedora_18_Final_Release_Criteria . I also moved the 'upgrade' criterion up a bit into what I think of as the 'install section' at the same time, so the change is a bit confused, sorry about that. (The criteria are roughly organized into component groups, though this isn't clearly called out, another deficiency of the current layout). -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: Criterion proposal: security
On 10/25/2012 09:16 PM, Adam Williamson wrote: Sure, that's the reason for the potential distinction between bugs that _can_ be fixed with updates and those that_can't_. This discussion was prompted by a specific bug - https://bugzilla.redhat.com/show_bug.cgi?id=868519 - which can't be fixed with an update, because it's an issue in anaconda. Just like any bug in anaconda, we can't fix it with a post-release update. For the first we could fix this with an post-release update but as you know certain people did not want that even if there was sufficient man power that was willing to maintain and do that from the community ( fedora-unity ) it although I agree with vincent in c14 this is not a bug as I read it this is working as they intend it... C3... This is working as we intend it. C5... kickstart does not allow for expressing encrypted passphrases, and we have no plans to add that. From my pov they simply should not store the password line et all. The bottom line is we already have an policy with regards to security updates which worst case scenario would be pulled in with an post-release 0 day update and security updates they themselves can bring a plethora of brokenness with them just like any other bug. To give you an example install of an another security issue we have been knowingly shipping for years off the dvd install sshd is enabled by default exposing it to bruteforce attacks immediately out of the box ( how long to think it's going to take with an cloud and if the host is on edubandwith ) while having no means of notifying the end user when it's happening. We allow that! Anaconda developers can always make the case since the file is owned and readable by root that the end user has an bigger issue to deal with if someone can access it ( classic case of security theater )... So now you want to create/tighten a security criteria because of an political debate with the maintainers and this hits what are we supposed to do if upstream does not provide security fixes for a particular release as I mentioned before. In the end of the day in the release process we need to treat security bugs as NTH as in we pull them if a) we know about b) it does not break anything c) cant be fixed with an 0 day update instead of delaying the release indefinitely ( best effort approach )... Personally I dont think we should have security release criteria to complicated the release process and that Anaconda implementation debate should be address by FESCO after all they exist for that exact purpose and they are the once that approved the newUI feature even when it was no way ready ( and still is not ) so they just get to eat their own dog food. JBG -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: Criterion proposal: security
On Fri, 2012-10-26 at 09:35 +, Jóhann B. Guðmundsson wrote: On 10/25/2012 09:16 PM, Adam Williamson wrote: Sure, that's the reason for the potential distinction between bugs that _can_ be fixed with updates and those that_can't_. This discussion was prompted by a specific bug - https://bugzilla.redhat.com/show_bug.cgi?id=868519 - which can't be fixed with an update, because it's an issue in anaconda. Just like any bug in anaconda, we can't fix it with a post-release update. For the first we could fix this with an post-release update but as you know certain people did not want that even if there was sufficient man power that was willing to maintain and do that from the community ( fedora-unity ) You're derailing again. The fact is that we don't release official updates of the install images and therefore bugs in the install path cannot be fixed via updates. If this changes then obviously we could consider changes to the release criteria and validation process, but right now, it's a *fact*: we do not release installer updates, therefore we have to maintain the distinction between bugs that can be fixed with updates and bugs that cannot. Discussion of whether we _ought_ to release updated installer images is fine and good, but it should be separate from this. it although I agree with vincent in c14 this is not a bug as I read it this is working as they intend it... C3... This is working as we intend it. C5... kickstart does not allow for expressing encrypted passphrases, and we have no plans to add that. From my pov they simply should not store the password line et all. The bottom line is we already have an policy with regards to security updates which worst case scenario would be pulled in with an post-release 0 day update and security updates they themselves can bring a plethora of brokenness with them just like any other bug. I don't think it really needs reiterating, but again, it is inarguably the case that it is possible for there to be security issues we cannot fix with updates. All code is potentially vulnerable to security issues, we ship code that cannot be changed by our update process, ergo there is a possibility of security issues we cannot fix with updates. QED. Specific examples of this are just examples. To give you an example install of an another security issue we have been knowingly shipping for years off the dvd install sshd is enabled by default exposing it to bruteforce attacks immediately out of the box ( how long to think it's going to take with an cloud and if the host is on edubandwith ) while having no means of notifying the end user when it's happening. We allow that! Again I don't really see the relevance to this debate. That has been discussed before and it's been decided that it does not constitute a security bug in Fedora as it's an intentional configuration choice where we made the tradeoff between convenience and security in the direction of security. Whenever you're building something as complex as a distribution there will be occasions where you have to decide between convenience and security; when these are properly considered and the decision is made on the side of convenience, that does not constitute a security bug. Even if you want to reanimate the question of whether this particular decision is correct, it really doesn't impact on the question of whether we should consider security bugs as blocker bugs in some circumstances, so far as I can see. It's just yet another tangent. Anaconda developers can always make the case since the file is owned and readable by root that the end user has an bigger issue to deal with if someone can access it ( classic case of security theater )... So now you want to create/tighten a security criteria because of an political debate with the maintainers and this hits what are we supposed to do if upstream does not provide security fixes for a particular release as I mentioned before. No, I think you're misrepresenting things there. There is a debate about whether this is actually a 'bug' or not, which really just means it's another case of what I mentioned above: we're considering an issue and deciding the trade-off between security and convenience. I am not proposing this criterion with the intent that, if it's accepted, I will post to the bug 'HAHA we just added a security criterion therefore this is a blocker bug and you can't argue!'. That's not the idea at all. Whether we decide to take security bugs as release blockers or not does not answer the question of _whether this is considered a 'bug' or not at all'. That would still be up for debate. But, theoretically speaking, if we were to accept that this was a security bug, then under the proposed criteria it may count as a blocker bug. Without any criteria it definitely wouldn't. I was proposing the criterion not because I want to use it as a stick to beat anaconda with, but simply because I wanted
Re: Criterion proposal: security
On 10/26/2012 07:14 PM, Adam Williamson wrote: I wanted to raise the question of whether it makes sense in general to hold our releases for some security bugs. Right now we have no capacity to do that. I dont think that should be for us to decide. When we encounter potential security issue in the development release cycle we should just forward those issue to the security team to determine if that's the case and let's assume it is then *they* would contact fesco which in turn decides if the release should be *delayed* or not until that security issue has been addressed. Given that these issue are few and far in between I dont think it warrants an specific criteria surrounding it but should rather be dealt on a case by case bases. The security community exists for this exact purpose so let's just let them handle that. They are expert in what they do... JBG -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: Criterion proposal: security
On Fri, 2012-10-26 at 19:33 +, Jóhann B. Guðmundsson wrote: On 10/26/2012 07:14 PM, Adam Williamson wrote: I wanted to raise the question of whether it makes sense in general to hold our releases for some security bugs. Right now we have no capacity to do that. I dont think that should be for us to decide. When we encounter potential security issue in the development release cycle we should just forward those issue to the security team to determine if that's the case and let's assume it is then *they* would contact fesco which in turn decides if the release should be *delayed* or not until that security issue has been addressed. Given that these issue are few and far in between I dont think it warrants an specific criteria surrounding it but should rather be dealt on a case by case bases. Oh, and in case this helps, I wasn't planning on adding a test case which says 'go test the entire distribution for security issues', or anything. The idea was just that this would be a criterion we would 'hold in reserve' to use when security issues were elevated to our attention. So really it just provides a mechanism for us to take a security issue that someone has raised that really seems to be a problem, and give it blocker status. I think with the feedback we've seen so far that we can say the original proposal was substantially too broad, so how about this as a revised proposal - for now, we just add a single Final release criterion which reads: The release must contain no known security issues of 'important' or higher impact according to the Red Hat severity classification scale which cannot be satisfactorily resolved by a package update (e.g. issues during installation) ? How does that sound to everyone? It drops the issue entirely for Alpha and Beta, and means we only consider bad issues that cannot be fixed with an update for Final. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: Criterion proposal: security
On 10/26/2012 07:44 PM, Adam Williamson wrote: The release must contain no known security issues of 'important' or higher impact according to the Red Hat severity classification scale which cannot be satisfactorily resolved by a package update (e.g. issues during installation) ? How does that sound to everyone? It drops the issue entirely for Alpha and Beta, and means we only consider bad issues that cannot be fixed with an update for Final. -- Ack -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: Criterion proposal: security
On Fri, Oct 26, 2012 at 12:44:24 -0700, Adam Williamson awill...@redhat.com wrote: ? How does that sound to everyone? It drops the issue entirely for Alpha and Beta, and means we only consider bad issues that cannot be fixed with an update for Final. That sounds reasonable. For alpha and beta we could still warn people if appropriate using common bugs. -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: Criterion proposal: security
On Fri, 2012-10-26 at 12:44 -0700, Adam Williamson wrote: I think with the feedback we've seen so far that we can say the original proposal was substantially too broad, so how about this as a revised proposal - for now, we just add a single Final release criterion which reads: The release must contain no known security issues of 'important' or higher impact according to the Red Hat severity classification scale which cannot be satisfactorily resolved by a package update (e.g. issues during installation) ? How does that sound to everyone? It drops the issue entirely for Alpha and Beta, and means we only consider bad issues that cannot be fixed with an update for Final. Hmm, actually, let's change 'issues' to 'bugs' there, I think that makes it clearer that we're talking about things that have actually been accepted as bugs - it avoids any suggestion we'd be wading into the debate about what actually constitutes a security issue, as Johann was concerned about. So: The release must contain no known security bugs of 'important' or higher impact according to the Red Hat severity classification scale which cannot be satisfactorily resolved by a package update (e.g. issues during installation) with the understanding that QA would never use this to wade into something like the sshd question and declare that it was a Bug That Must Be Fixed. It applies only to things that are clearly agreed to be actual bugs. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: Criterion proposal: security
On Fri, 2012-10-26 at 19:33 +, Jóhann B. Guðmundsson wrote: On 10/26/2012 07:14 PM, Adam Williamson wrote: I wanted to raise the question of whether it makes sense in general to hold our releases for some security bugs. Right now we have no capacity to do that. I dont think that should be for us to decide. When we encounter potential security issue in the development release cycle we should just forward those issue to the security team to determine if that's the case and let's assume it is then *they* would contact fesco which in turn decides if the release should be *delayed* or not until that security issue has been addressed. Given that these issue are few and far in between I dont think it warrants an specific criteria surrounding it but should rather be dealt on a case by case bases. The security community exists for this exact purpose so let's just let them handle that. They are expert in what they do... Well, Vincent seemed to think it would be good to handle this as a matter of policy via the blocker process and not case-by-case by FESCo. I don't think it's quite like the feature process case where we say 'feature issues go to FESCo not through the blocker process', because the feature process is a Thing that Exists and is owned by FESCo. We don't _have_ a security bug process at present, really. At least so far as blocking releases is concerned. Anything introduced at this point would be an invention, whether it goes via FESCo or the release validation process. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: Criterion proposal: security
On 10/25/2012 12:06 AM, Adam Williamson wrote: What do folks think of this? Anyone want to tweak what severity issues we block for when, or think the approach is bad? Thanks! We might not want to start at Alpha, on the basis that Alpha is supposed to be for testing*only* and should never have sensitive data committed to it, for instance: we could combine the proposed Alpha criterion into the Beta one. I'm not so sure we want to add security related clause to the criteria since security flaws are just bugs but if people insist that we add one I think we should only apply that security criteria only to final so we wont release the distribution with *known* security fiaw. JBG -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: Criterion proposal: security
On Thu, 2012-10-25 at 10:17 +, Jóhann B. Guðmundsson wrote: On 10/25/2012 12:06 AM, Adam Williamson wrote: What do folks think of this? Anyone want to tweak what severity issues we block for when, or think the approach is bad? Thanks! We might not want to start at Alpha, on the basis that Alpha is supposed to be for testing *only* and should never have sensitive data committed to it, for instance: we could combine the proposed Alpha criterion into the Beta one. I'm not so sure we want to add security related clause to the criteria since security flaws are just bugs I'm not quite sure what you mean...all blocker bugs are 'just bugs', some bugs are important enough that we shouldn't make releases with those bugs in them. We can't really claim that security bugs violate any of the existing criteria because the existing criteria are focused on functionality, not security. A flaw which would let anyone on the internet read all your data does not, so far as I can see, violate any of the existing criteria (unless you use the 'escape clause' of 'any high severity bug in a critpath package is a blocker, but in practice, we have never really applied that). I think it should constitute a blocker...if others agree, then we need a criterion to express that, since we don't currently have one. but if people insist that we add one I think we should only apply that security criteria only to final so we wont release the distribution with *known* security fiaw. Yeah, I did consider that; it's a viable line to say that Alpha and Beta are test releases so security flaws are not really that significant, you should be using them only for test purposes and trusting no valuable data / systems to them. I think that may well be plausible for Alpha. For Beta it's a bit more problematic, because in practice, 'testing a Beta' can involve giving it at least some exposure to sensitive data/processes. Say you run Fedora on 1,000 client boxes (I don't know if anyone does, but just supposing...) at your school/enterprise/whatever, with a central server containing important data. I think 'testing' Fedora XX Beta is probably going to involve deploying it on one 'test' client in your setup...which is probably going to interact with your actual 'production' network in some way. So I think it might be hard to maintain that we don't need to care about security at all for the Beta. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: Criterion proposal: security
On 10/25/2012 06:47 PM, Adam Williamson wrote: On Thu, 2012-10-25 at 10:17 +, Jóhann B. Guðmundsson wrote: On 10/25/2012 12:06 AM, Adam Williamson wrote: What do folks think of this? Anyone want to tweak what severity issues we block for when, or think the approach is bad? Thanks! We might not want to start at Alpha, on the basis that Alpha is supposed to be for testing *only* and should never have sensitive data committed to it, for instance: we could combine the proposed Alpha criterion into the Beta one. I'm not so sure we want to add security related clause to the criteria since security flaws are just bugs I'm not quite sure what you mean...all blocker bugs are 'just bugs', some bugs are important enough that we shouldn't make releases with those bugs in them. We can't really claim that security bugs violate any of the existing criteria because the existing criteria are focused on functionality, not security. A flaw which would let anyone on the internet read all your data does not, so far as I can see, violate any of the existing criteria (unless you use the 'escape clause' of 'any high severity bug in a critpath package is a blocker, but in practice, we have never really applied that). I think it should constitute a blocker...if others agree, then we need a criterion to express that, since we don't currently have one. but if people insist that we add one I think we should only apply that security criteria only to final so we wont release the distribution with *known* security fiaw. Yeah, I did consider that; it's a viable line to say that Alpha and Beta are test releases so security flaws are not really that significant, you should be using them only for test purposes and trusting no valuable data / systems to them. I think that may well be plausible for Alpha. For Beta it's a bit more problematic, because in practice, 'testing a Beta' can involve giving it at least some exposure to sensitive data/processes. Say you run Fedora on 1,000 client boxes (I don't know if anyone does, but just supposing...) at your school/enterprise/whatever, with a central server containing important data. I think 'testing' Fedora XX Beta is probably going to involve deploying it on one 'test' client in your setup...which is probably going to interact with your actual 'production' network in some way. So I think it might be hard to maintain that we don't need to care about security at all for the Beta. Beside the obvious point that a regular bug risk doing just that as well and we already have an policy that is in place to deal with security updates in general, what are we supposed to do if upstream does not provide security fixes for a particular release, and if backporting the fix would be impractical,delay the release indefinitely until they do? I've cc the Releng community to get their input on this + I think we discuss and decide to stay away from security related criteria in the past. JBG -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: Criterion proposal: security
On Thu, 2012-10-25 at 19:11 +, Jóhann B. Guðmundsson wrote: but if people insist that we add one I think we should only apply that security criteria only to final so we wont release the distribution with *known* security fiaw. Yeah, I did consider that; it's a viable line to say that Alpha and Beta are test releases so security flaws are not really that significant, you should be using them only for test purposes and trusting no valuable data / systems to them. I think that may well be plausible for Alpha. For Beta it's a bit more problematic, because in practice, 'testing a Beta' can involve giving it at least some exposure to sensitive data/processes. Say you run Fedora on 1,000 client boxes (I don't know if anyone does, but just supposing...) at your school/enterprise/whatever, with a central server containing important data. I think 'testing' Fedora XX Beta is probably going to involve deploying it on one 'test' client in your setup...which is probably going to interact with your actual 'production' network in some way. So I think it might be hard to maintain that we don't need to care about security at all for the Beta. Beside the obvious point that a regular bug risk doing just that as well and we already have an policy that is in place to deal with security updates in general, Sure, that's the reason for the potential distinction between bugs that _can_ be fixed with updates and those that _can't_. This discussion was prompted by a specific bug - https://bugzilla.redhat.com/show_bug.cgi?id=868519 - which can't be fixed with an update, because it's an issue in anaconda. Just like any bug in anaconda, we can't fix it with a post-release update. what are we supposed to do if upstream does not provide security fixes for a particular release, and if backporting the fix would be impractical,delay the release indefinitely until they do? That is a valid concern, indeed. We could add wording to exclude bugs for which a fix is not viable? Also if we restrict the criteria to only security bugs that are not fixable with an update, that might go quite some way to addressing this in practice, because I think it's fair to say we ought to have the development resources within Fedora to come up for a fix to any 'non-update-fixable' security issues ourselves. I've cc the Releng community to get their input on this + I think we discuss and decide to stay away from security related criteria in the past. Thanks, if anyone has useful data from past discussions it'd help indeed. In my archives I can find *part* of a thread from May 2011 but I can't find the start of it...I'll have a closer look later. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: Criterion proposal: security
On Thu, 2012-10-25 at 21:57 -0600, Vincent Danen wrote: So whether that's done earlier in the cycle or later, it doesn't matter. Earlier on gives it more time to get fixed. The end result is the same: it has to be fixed. I don't see why making it a blocker in the alpha stage vs making it a blocker in the pre-GA stage makes a difference other than you're giving a developer less time to fix something that absolutely needs to be fixed. Well...not really. If a bug is an Alpha blocker it has to be fixed by Alpha. If a bug is a Final blocker it has to be fixed by Final. So if you start at a theoretical point one week before the Alpha gets signed off, you have one week to fix the bug if it's an Alpha blocker. You have, oh, about three months and one week to fix the bug if it's a Final blocker. I still don't understand the RHEL process entirely but it seems to involve considering blocker status more or less independent of milestone, so you might put 'blocker+' on a bug and have the target release (or whatever it's called) as 'alpha', but that doesn't really mean it's an Alpha blocker - maybe one week before the Alpha goes out, half the bugs get the target release changed to 'beta' and the Alpha goes out anyway. Again, we don't do that for Fedora. We have three blocker lists per release; Alpha blockers, Beta blockers, Final blockers. We don't take a look at the Alpha blocker list one week before Alpha release and go 'well hmm, that looks a bit long, maybe we should bump some of them to Beta'. So we have weaker criteria for Alpha and Beta because, well, Alpha and Beta release don't have to be as stable as Final releases. Given the tight schedules of Fedora we can't just take the Final standards and apply them to Alpha. We just don't have time. The Alpha criteria are very minimal and define only what absolutely has to be done for an Alpha to go out. If you look at the criteria, we can release Alphas with gigantic flaws in them. If you think it wouldn't realistic to say 'we'll take any persistent moderate flaw as a blocker' that's valuable feedback...then the question becomes do we only bother about important+ bugs, do we try and come up with some kind of solid definition for what 'moderate' bugs we'll consider blockers and which ones we won't, or do we just say 'we'll evaluate 'moderate' bugs on a case-by-case basis'. As stated above, I think you can break down the moderate flaws into types. If you do that, you should rarely have to look at exceptions, I think. I mean, if there is a really bad local flaw that wouldn't fit into the category of a what is defined as a moderate release blocker, but it's really that bad, then chances are it's not classified correctly and may actually be an important flaw. CVSSv2 can help here. So can the SRT, or the Fedora Security Response Team (many of whom are SRT too). A simple what do you think sent to either team will get someone with experience to evaluate the flaw. That sounds like a good approach, thanks. Yup, that's useful feedback indeed, and thanks. I think a policy would be extremely helpful as it takes the guess-work out of things. Definitely, that's exactly the idea. I'm also appreciative that you reached out for feedback because there is a lot of assistance that we (SRT) would _like_ to give to Fedora, and we're doing as much as we can, but this kind of outreach is really quite nice and we're more than happy to lend any kind of assistance that we can. Conversations for another day. I would be happy with this as a first start, honestly. A cornerstone to start building a more comprehensive policy on is nothing to be taken lightly. =) Thanks for bringing this up, Adam. Thanks a lot for the input! You're very welcome. I hope it's helpful! Please do let me (or the SRT in general) know if you need any further help, even if helping to define what a moderate release blocker might look like. Again, we are more than happy to help. I'll look at the points you and other responders raised and try to come up with tweaked criteria tomorrow. Thanks! -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Criterion proposal: security
So, IIRC, we've vaguely considered this before, but never really come up with a criterion proposal. But I think we need one or two, to explicitly allow security issues to be blockers. I *really* don't want to get into the business of defining what constitutes a security issue, and what the severity of various types of issue is, in the criteria. It's a complex and specialized area, and there are already authorities for doing this. I just had a look at the major ones - notably CVSS - and it's crazy complex and we're not likely to be doing those calculations ourselves, so I'm thinking it may be best to take advantage of the relatively simple Red Hat security classifications: https://access.redhat.com/security/updates/classification/ That would allow us to say something like this: Alpha: The release must contain no known security issues of 'critical' impact according to the Red Hat severity classification scale Beta: The release must contain no known security issues of 'important' impact according to the Red Hat severity classification scale which cannot be fully resolved by a package update (e.g. issues during installation) Final: The release must contain no known security issues of 'important' impact according to the Red Hat severity classification scale, or issues of 'moderate' impact which cannot be fully resolved by a package update (e.g. issues during installation) (remember that criteria are additive, the Alpha criterion would apply to Beta and Final) Something like that - the precise boundaries we could tweak, but I think the basic idea flies. We use the severities from the RH classification and we draw a line between issues that could be fixed with an update (most will fall in this category) and ones we can't fix with an update - like issues in anaconda. What do folks think of this? Anyone want to tweak what severity issues we block for when, or think the approach is bad? Thanks! We might not want to start at Alpha, on the basis that Alpha is supposed to be for testing *only* and should never have sensitive data committed to it, for instance: we could combine the proposed Alpha criterion into the Beta one. CCing vdanen (from the security team) for a security expert perspective. Vincent, your input would be appreciated, remember we can't set bars *too* high for Fedora due to the release cycle and available development resources - it'd be interesting to know what rules RHEL uses here, but we can't necessarily do the same for Fedora. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: Criterion proposal: security
On Wed, Oct 24, 2012 at 05:06:41PM -0700, Adam Williamson wrote: What do folks think of this? Anyone want to tweak what severity issues +1 we block for when, or think the approach is bad? Thanks! We might not want to start at Alpha, on the basis that Alpha is supposed to be for testing *only* and should never have sensitive data committed to it, for instance: we could combine the proposed Alpha criterion into the Beta one. critical is pretty serious on the Red Hat scale. I think the way you have it is reasonable. -- Matthew Miller ☁☁☁ Fedora Cloud Architect ☁☁☁ mat...@fedoraproject.org -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: Criterion proposal: security
On Wed, 2012-10-24 at 19:38 -0600, Vincent Danen wrote: So if you're looking at alpha's as having all criticals fixed, I think you can say that is quite well understood and I don't think it's a problem. Important-impact issues in beta and release is totally doable. It's the lower-impact stuff I'd like to see resolved properly. For this thread I want to keep it to what goes in the release criteria, if we can. If we spread the discussion too wide we'll never get done. Note that, if I'm understanding your mail correct, what you call 'persistent flaws' is what I'm referring to when I talk about 'cannot be fully resolved by a package update' - we're talking about stuff in the install process or live boot or first boot, which we can't resolve just by shipping an update. That's why I made that distinction and gave those bugs, in general, more importance. What's your opinion on moderate bugs that can't be fixed by updates (i.e. persistent flaws) as a Final blocker? Is that going too far? Note the blocker process is pretty much orthogonal to priority/severity. A blocker issue for Fedora is a blocker. We don't ship the release without fixing it. It's a very simple binary state. It doesn't matter what priority/severity a blocker bug happens to get assigned, it must be fixed for the release to go out. Our rule for RHEL is no known important+ security flaws. This may lead to a bunch of 0-day errata to get things squared away at GA (if we find out about something post-freeze, etc.). This is for dot releases, like 6.2 or 6.3. For a new dot-0 release the rules are different. For those releases, we absolutely strive for no known security flaws of _any_ impact. We don't want to release a product with known security issues. How does that equate to Fedora? I don't know. Do we look at Fedora's quick release cycle as a series of .X releases? Or are they all .0 releases? I'd love to see them as .0 with no known security issues, but I suspect that won't happen anytime soon. =) Yeah, that would be too ambitious. The other difference in Fedora is we don't consider 0-day updates part of a release. We ship a bunch of 0-days, but they're just not considered in the 'release process', except in very special circumstances. Blocker bugs have to be fixed *in the release package set* (what gets frozen). To me, no known important+ flaws at a Fedora GA is a doable goal, and I'm pretty sure that more often than not we achieve that without really aiming for it. Making this into some sort of policy makes sure it's followed, but I think the real benefactor to this policy would be Anaconda. Really it's just about getting things into the policy. 'Achieving the goal without really aiming for it' is fine and all, but it's not bulletproof :) There've been a few cases where we've kinda wanted to take security bugs as blockers but simply haven't had a framework for it. We're very strict on the blocker process these days: we don't just throw the 'blocker' status around arbitrarily. There _has_ to be a justification for it in the criteria. I'd like to see some kind of improved policy for dealing with security bugs in Fedora in general, but I think that is a resource problem and the fact that the Fedora Security Response Team really doesn't do anything. Most of the duties that it would have, Red Hat Security Response handles. The real hole in our process is we don't followup/badger/whatever once these tracking bugs are filed. We tend to fire and forget, which may be the wrong approach, but is really the only one that works for us due to time/resource constraints. Perhaps the Fedora Security Response Team could be more formalized and chase those things? Maybe that's a good first step to aim for? That might be outside the scope of what you're bringing up here, Adam. Sorry if you feel like maybe you opened a can of worms. You just gave me a soap-box I've been eyeing for a few years. =) Yes, I think we're somewhat out of scope here. If you want to start up that discussion then go for it, but I'd _really_ like to keep criterion proposal threads on track, because they're not talking shops. Criteria revisions are important actions in QA, this isn't just a trial balloon, the intent is to put something in place in a fairly short term here. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
Re: Criterion proposal: security
On Wed, 2012-10-24 at 20:10 -0600, Vincent Danen wrote: What's your opinion on moderate bugs that can't be fixed by updates (i.e. persistent flaws) as a Final blocker? Is that going too far? I don't think that's going too far, but again it should be evaluated on a case-by-case basis. Some moderates are more serious than others. We often fix moderate flaws in errata, but there are plenty that we defer. They are, by classification, moderate but just aren't worth the effort to do a fix for it right now. I think the same could be done here. Important+: no question. Moderate: maybe it needs to get some acks before it is considered a blocker? What we're trying to do with the criteria is reduce the occurrence of 'case-by-case evaluation' as much as we can. It's inevitably the case that we can't entirely eliminate it, but we want to make things as objective as we can. For instance, a moderate flaw that only would affect a small subset of users probably shouldn't be a release blocker if it's found a week before GA. One that affects 90% of users probably should. Again, I don't know the process here, but maybe it would make sense to have moderate flaws block the alpha (and maybe the beta?) but not later. This gives you an out -- two weeks to release and someone finds a moderate flaw, you don't have to necessarily delay the release. But if it's found a month before GA and can be fixed prior to the alpha release, maybe that should be considered just so that it gets done. (My fear otherwise is that if you have critical-only as blocking alpha, you _have_ to leave it to a later stage that might be more inconvenient). Ah, the old RHEL attitude again ;) Funny how things come in twos. This seems to be a RHEL strategy - throw everything but the kitchen sink on the 'blocker' list early in the process, and get tighter closer to release. That's not how we do things in Fedora, we do things the other way around, really. Note that in Fedora the blocker list is not a todo list or a 'let's not forget about these bugs' list. It is a _list of bugs that block the release_. That's all it is. No bug is ever intended to be on the list of blockers for a given release unless it's really and truly blocking that release. We don't use the blocker list for 'just so that it gets done' purposes. We decided a few years back that that way of doing things kind of sucked, and designed a more rigorous way of doing things :) If you think it wouldn't realistic to say 'we'll take any persistent moderate flaw as a blocker' that's valuable feedback...then the question becomes do we only bother about important+ bugs, do we try and come up with some kind of solid definition for what 'moderate' bugs we'll consider blockers and which ones we won't, or do we just say 'we'll evaluate 'moderate' bugs on a case-by-case basis'. Yeah, that would be too ambitious. The other difference in Fedora is we don't consider 0-day updates part of a release. We ship a bunch of 0-days, but they're just not considered in the 'release process', except in very special circumstances. Blocker bugs have to be fixed *in the release package set* (what gets frozen). That makes sense. I didn't think that Fedora had 0-day errata like we do with RHEL releases. They just get pumped out on or after release date, regardless of security-relevance (at least that's what it seems to me, but as far as Fedora is concerned, I'm just an end-user). The way you say it it does sound like there's a difference, because for Fedora, a 0-day update is just any update that happens to have been pushed stable by the official release date. There's no special handling of such updates, they just happen. What actually happens is that we lift freeze after the go/no-go meeting but before release; any update that accumulates sufficient karma and gets pushed 'stable' in the period between the freeze being lifted and the release going out becomes a 0-day update, i.e., it's available on release day. There's no special shepherding of such updates, usually. Absolutely. I didn't mean to imply you shouldn't make it a policy because it's more often than not a non-issue. Rather, I meant it should be easy enough to get into place as a policy because it's been easily achieved (so far), so can't really be seen as a major undertaking or too much to ask for kind of thing. Yup, that's useful feedback indeed, and thanks. Conversations for another day. I would be happy with this as a first start, honestly. A cornerstone to start building a more comprehensive policy on is nothing to be taken lightly. =) Thanks for bringing this up, Adam. Thanks a lot for the input! -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
