Re: [Xen-devel] Xen Project Security Whitepaper v1 is ready for community review
>>> On 24.05.18 at 16:14, wrote: > How many bug-fixes vs. XSAs are typically in a stable branch? I was under > the impression that historically, the vast majority used to be XSAs with very > few backports. > However, this year this has really changed because Spectre and Meltdown > related fixes were developed in public and they look like feature backports > Which is why we see more of these issues The ratio may vary, but I think it has always been more non-security than security fixes, at least for as long as I've been stable branch maintainer. It otherwise also wouldn't make much sense to distinguish between fully maintained branches and ones in security-only maintenance mode. Jan ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] Xen Project Security Whitepaper v1 is ready for community review
On 24/05/2018, 10:00, "Jan Beulich" wrote: >>> On 24.05.18 at 15:50, wrote: > > On 24/05/2018, 01:48, "Steven Haigh" wrote: > > On 2018-05-22 20:52, Steven Haigh wrote: > > On Tuesday, 22 May 2018 8:11:38 PM AEST Jan Beulich wrote: > >> >>> On 18.05.18 at 19:53, wrote: > >> > Alternative workaround for this would be more frequent point releases > by > >> > default (maybe with ability to delay it very few commits are queued). > >> > For example every 3 months. It wouldn't solve all the cases, but I > think > >> > will make it easier most of the time. > >> > >> Is every 3 months so much better than every 4 months? Granted we > >> basically never manage to make it exactly 4 months, but on the average > >> I think we're not too far off. > > > > I think the big thing is reducing the delta between the staging branch > > and the > > release. I can only assume that would reduce the number of issues that > > occur > > with patching vs release tarballs - hopefully making the security teams > > > job a > > little easier. > > > > That being said, if an approach of releasing a new build when we come > > across > > broken patch sets for XSAs (like the current 4.9.1 vs XSAs, and prior > > 4.10.0 > > vs XSAs), then I think this part becomes irrelevant. > > As another example for this, the patches for XSA263 do not apply to > *any* released tarball version of Xen. > > So far, the patches included with the announcement fail on 4.6, 4.7, 4.9 > > and 4.10. > > I can only assume that this means all the XSA patches require commits > that are currently in various staging git trees that have not been > released in any formal manner via a point release. > > Thinking about this, we are essentially exposing ourselves to this, because > of backports of issues which happen at any random point in time during a > point release cycle, e.g. 4.8.2. => 4.8.3 > > In other words, we may get a sequence of backport, XSA, XSA, backport, ... > > I am wondering whether time-sequencing may be the answer here. In other > words, let's assume we have a 4-month window: for the first 3 months, we > don't allow bug-fix backports into in this case stable-4.8, we only allow > XSAs. Then we have a 2-week merge window where we handle all backports and > prepare the release and cut the release. This means that for most of the > time, XSAs would apply cleanly onto staging and the released tarball. I'm sincerely against such a model: The larger a batch of commits, the more likely that some then-no-longer-easy-to-spot regression may creep in, and the less time there is for osstest and people to test. I do appreciate some batching, but just like with XSAs I think the batch size should remain sensible also for commits to stable trees. How many bug-fixes vs. XSAs are typically in a stable branch? I was under the impression that historically, the vast majority used to be XSAs with very few backports. However, this year this has really changed because Spectre and Meltdown related fixes were developed in public and they look like feature backports Which is why we see more of these issues Lars ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] Xen Project Security Whitepaper v1 is ready for community review
>>> On 24.05.18 at 15:50, wrote: > > On 24/05/2018, 01:48, "Steven Haigh" wrote: > > On 2018-05-22 20:52, Steven Haigh wrote: > > On Tuesday, 22 May 2018 8:11:38 PM AEST Jan Beulich wrote: > >> >>> On 18.05.18 at 19:53, wrote: > >> > Alternative workaround for this would be more frequent point > releases > by > >> > default (maybe with ability to delay it very few commits are queued). > >> > For example every 3 months. It wouldn't solve all the cases, but I > think > >> > will make it easier most of the time. > >> > >> Is every 3 months so much better than every 4 months? Granted we > >> basically never manage to make it exactly 4 months, but on the average > >> I think we're not too far off. > > > > I think the big thing is reducing the delta between the staging branch > > and the > > release. I can only assume that would reduce the number of issues that > > occur > > with patching vs release tarballs - hopefully making the security teams > > > job a > > little easier. > > > > That being said, if an approach of releasing a new build when we come > > across > > broken patch sets for XSAs (like the current 4.9.1 vs XSAs, and prior > > 4.10.0 > > vs XSAs), then I think this part becomes irrelevant. > > As another example for this, the patches for XSA263 do not apply to > *any* released tarball version of Xen. > > So far, the patches included with the announcement fail on 4.6, 4.7, 4.9 > > and 4.10. > > I can only assume that this means all the XSA patches require commits > that are currently in various staging git trees that have not been > released in any formal manner via a point release. > > Thinking about this, we are essentially exposing ourselves to this, because > of backports of issues which happen at any random point in time during a > point release cycle, e.g. 4.8.2. => 4.8.3 > > In other words, we may get a sequence of backport, XSA, XSA, backport, ... > > I am wondering whether time-sequencing may be the answer here. In other > words, let's assume we have a 4-month window: for the first 3 months, we > don't allow bug-fix backports into in this case stable-4.8, we only allow > XSAs. Then we have a 2-week merge window where we handle all backports and > prepare the release and cut the release. This means that for most of the > time, XSAs would apply cleanly onto staging and the released tarball. I'm sincerely against such a model: The larger a batch of commits, the more likely that some then-no-longer-easy-to-spot regression may creep in, and the less time there is for osstest and people to test. I do appreciate some batching, but just like with XSAs I think the batch size should remain sensible also for commits to stable trees. I can see that in particular for XSA-263 it would have been useful if we had enumerated the prereqs (it was in fact intentional that we had decided to commit some of them ahead of time, so that the advisory itself would be more manageable in size and otherwise). But face it - compiling such a list does add non-negligible extra work on the security team's part. Jan ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] Xen Project Security Whitepaper v1 is ready for community review
On 24/05/2018, 01:48, "Steven Haigh" wrote: On 2018-05-22 20:52, Steven Haigh wrote: > On Tuesday, 22 May 2018 8:11:38 PM AEST Jan Beulich wrote: >> >>> On 18.05.18 at 19:53, wrote: >> > Alternative workaround for this would be more frequent point releases by >> > default (maybe with ability to delay it very few commits are queued). >> > For example every 3 months. It wouldn't solve all the cases, but I think >> > will make it easier most of the time. >> >> Is every 3 months so much better than every 4 months? Granted we >> basically never manage to make it exactly 4 months, but on the average >> I think we're not too far off. > > I think the big thing is reducing the delta between the staging branch > and the > release. I can only assume that would reduce the number of issues that > occur > with patching vs release tarballs - hopefully making the security teams > job a > little easier. > > That being said, if an approach of releasing a new build when we come > across > broken patch sets for XSAs (like the current 4.9.1 vs XSAs, and prior > 4.10.0 > vs XSAs), then I think this part becomes irrelevant. As another example for this, the patches for XSA263 do not apply to *any* released tarball version of Xen. So far, the patches included with the announcement fail on 4.6, 4.7, 4.9 and 4.10. I can only assume that this means all the XSA patches require commits that are currently in various staging git trees that have not been released in any formal manner via a point release. Thinking about this, we are essentially exposing ourselves to this, because of backports of issues which happen at any random point in time during a point release cycle, e.g. 4.8.2. => 4.8.3 In other words, we may get a sequence of backport, XSA, XSA, backport, ... I am wondering whether time-sequencing may be the answer here. In other words, let's assume we have a 4-month window: for the first 3 months, we don't allow bug-fix backports into in this case stable-4.8, we only allow XSAs. Then we have a 2-week merge window where we handle all backports and prepare the release and cut the release. This means that for most of the time, XSAs would apply cleanly onto staging and the released tarball. If XSAs happen while we prepare the next week merge window for backports for a release (in this example 4.8). There may be some pain, if XSAs turn up during the merge window but we would release the next point release shortly afterwards, which means that users have the choice of a) go through the pain of cherry-picking (or applying all of the patches that come through in the 2 week merge window) - that should be easier as than now, because they would be more easily identifiable b) can just wait for the release and then apply XSAs again That shouldn't really create any extra work for point release maintainers, while reducing issues with patches not applying onto released tarballs I may have missed something here, and have not fully thought this through, but it may be worthwhile discussing further along those lines Cheers Lars ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] Xen Project Security Whitepaper v1 is ready for community review
On 2018-05-22 20:52, Steven Haigh wrote: On Tuesday, 22 May 2018 8:11:38 PM AEST Jan Beulich wrote: >>> On 18.05.18 at 19:53, wrote: > Alternative workaround for this would be more frequent point releases by > default (maybe with ability to delay it very few commits are queued). > For example every 3 months. It wouldn't solve all the cases, but I think > will make it easier most of the time. Is every 3 months so much better than every 4 months? Granted we basically never manage to make it exactly 4 months, but on the average I think we're not too far off. I think the big thing is reducing the delta between the staging branch and the release. I can only assume that would reduce the number of issues that occur with patching vs release tarballs - hopefully making the security teams job a little easier. That being said, if an approach of releasing a new build when we come across broken patch sets for XSAs (like the current 4.9.1 vs XSAs, and prior 4.10.0 vs XSAs), then I think this part becomes irrelevant. As another example for this, the patches for XSA263 do not apply to *any* released tarball version of Xen. So far, the patches included with the announcement fail on 4.6, 4.7, 4.9 and 4.10. I can only assume that this means all the XSA patches require commits that are currently in various staging git trees that have not been released in any formal manner via a point release. -- Steven Haigh ? net...@crc.id.au ? https://www.crc.id.au ? +61 (3) 9001 6090? 0412 935 897 ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] Xen Project Security Whitepaper v1 is ready for community review
On Tuesday, 22 May 2018 8:11:38 PM AEST Jan Beulich wrote: > >>> On 18.05.18 at 19:53, wrote: > > Alternative workaround for this would be more frequent point releases by > > default (maybe with ability to delay it very few commits are queued). > > For example every 3 months. It wouldn't solve all the cases, but I think > > will make it easier most of the time. > > Is every 3 months so much better than every 4 months? Granted we > basically never manage to make it exactly 4 months, but on the average > I think we're not too far off. I think the big thing is reducing the delta between the staging branch and the release. I can only assume that would reduce the number of issues that occur with patching vs release tarballs - hopefully making the security teams job a little easier. That being said, if an approach of releasing a new build when we come across broken patch sets for XSAs (like the current 4.9.1 vs XSAs, and prior 4.10.0 vs XSAs), then I think this part becomes irrelevant. -- Steven Haigh 📧 net...@crc.id.au 💻 https://www.crc.id.au 📞 +61 (3) 9001 6090📱 0412 935 897 signature.asc Description: This is a digitally signed message part. ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] Xen Project Security Whitepaper v1 is ready for community review
>>> On 18.05.18 at 19:53, wrote: > Alternative workaround for this would be more frequent point releases by > default (maybe with ability to delay it very few commits are queued). > For example every 3 months. It wouldn't solve all the cases, but I think > will make it easier most of the time. Is every 3 months so much better than every 4 months? Granted we basically never manage to make it exactly 4 months, but on the average I think we're not too far off. Jan ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] Xen Project Security Whitepaper v1 is ready for community review
On Sat, May 19, 2018 at 01:55:53AM +1000, Steven Haigh wrote: > Hi Lars, > > I think this is an excellent start. > > A specific concern that I have is when we get into a state between releases > and XSAs where you cannot take the current release and then apply all > released > / embargo'ed XSA patches. > > The current reasoning for this is that XSA patches are developed on top of > the > staging git branches. While this is still acceptable, I believe we need the > ability to roll a new point release that will allow end users to be up to > date. > > Expecting things to always be built for distribution from the staging git > branch is somewhat of a hassle - as in the current case of 4.9.1. With > publicly released XSAs, you cannot begin with a release of 4.9.1 and patch > all > post-released XSAs. > > While this does not seem to happen very often - I would estimate around 4-5 > times in the past decade - we should encourage an out-of-schedule point > release. This can be based off the current state post-XSA of the staging > branch - but enables reproducable builds at the very least. > > Recently, this situation happened with the batch of XSAs before 4.10.1 was > released, and is currently the case of 4.9.1 + existing XSAs. > > This potentially leaves end users in limbo until the next point release rolls > around - without rebasing off a semi-random git commit (which is not 4.9.1 or > 4.9.2 - but something inbetween) - or backporting massive amounts of commits > to a release. > > As this is a somewhat rare occasion, if only a handful of commits need to be > cherrypicked, I would see this as fine. If it requires many more, I believe > it > should trigger an out-of-cycle point release. I second all of the above. Having to figure out prerequisite patches (or adjusting patches to be applicable over the last point release) was an issue in the past multiple times. In some cases it isn't such a big issue, but there are also difficult ones. Alternative workaround for this would be more frequent point releases by default (maybe with ability to delay it very few commits are queued). For example every 3 months. It wouldn't solve all the cases, but I think will make it easier most of the time. As for the other points: 2.2.4 / 3.2 IMO 9-month release cycle would make sense. The current one is also an issue for us, as we freeze Xen major version multiple months before the release, so the short release means we're already behind at the release time. This affect for example backporting patches - like with Meltdown, patches for older versions were available with the delay. 3.1. R2 I mostly agree with Jan here. For Qubes OS, 2-week pre-disclosure time is enough even for large batches, and I'd rather avoid artificially delaying XSA, especially those with major impact (whatever definition would be used). But I see this might be an issue for other XSA consumers, so some flexibility for the Xen Security Team here would be fine. BTW The solution 1/2 comparison table have "Public releases by downstreams" row swapped - solution 1 imply 2 public releases. And I think the same in "Per batch cost". 3.1. R3 Fixed schedule indeed would help with some of the issues, so I'm slightly for it. -- Best Regards, Marek Marczykowski-Górecki Invisible Things Lab A: Because it messes up the order in which people normally read text. Q: Why is top-posting such a bad thing? signature.asc Description: PGP signature ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] Xen Project Security Whitepaper v1 is ready for community review
Hi Lars, I think this is an excellent start. A specific concern that I have is when we get into a state between releases and XSAs where you cannot take the current release and then apply all released / embargo'ed XSA patches. The current reasoning for this is that XSA patches are developed on top of the staging git branches. While this is still acceptable, I believe we need the ability to roll a new point release that will allow end users to be up to date. Expecting things to always be built for distribution from the staging git branch is somewhat of a hassle - as in the current case of 4.9.1. With publicly released XSAs, you cannot begin with a release of 4.9.1 and patch all post-released XSAs. While this does not seem to happen very often - I would estimate around 4-5 times in the past decade - we should encourage an out-of-schedule point release. This can be based off the current state post-XSA of the staging branch - but enables reproducable builds at the very least. Recently, this situation happened with the batch of XSAs before 4.10.1 was released, and is currently the case of 4.9.1 + existing XSAs. This potentially leaves end users in limbo until the next point release rolls around - without rebasing off a semi-random git commit (which is not 4.9.1 or 4.9.2 - but something inbetween) - or backporting massive amounts of commits to a release. As this is a somewhat rare occasion, if only a handful of commits need to be cherrypicked, I would see this as fine. If it requires many more, I believe it should trigger an out-of-cycle point release. -- Steven Haigh 📧 net...@crc.id.au 💻 https://www.crc.id.au 📞 +61 (3) 9001 6090📱 0412 935 897 On Friday, 18 May 2018 8:13:55 PM AEST Lars Kurth wrote: > Dear Community Members, > > just under 3 months ago, we started a community consultation titled "Xen > Security Process Consultation: is there a case to change anything?" (see > https://lists.xenproject.org/archives/html/xen-announce/2018-02/msg0.ht > ml). As promised, I would collate the input - together with further analysis > trying to genuinely consider the implications of what respondents to the > consultation have been suggesting - in a white paper. The white paper is > attached and contains > 1) Baseline: an analysis of our XSAs and how we dealt with XSAs in the > recent past 2) Results from the Community Consultation > 2.1) Feedback received from a community consultation > 2.2) Analysis > 3) Recommendations and policy changes - some is quite extensive to try and > tries to evaluate the impact of policy changes, which would result if we > implemented solutions to issues highlighted by our users. > The next step is for community members to provide public feedback. If it > turns out there is a case for changes/improvements, I will condense the > output of this discussion into a concrete change proposal (or a series > thereof) to be voted on in the usual way. This may require several > iterations. Note that the document contains workflow and tools related > feedback, which I did not anticipate. Some issues highlighted should be > easy to fix, others will require additional discussion on xen-devel@, such > as * Inconsistent Meta Data and XSA prerequisites > * Git baseline of patches > * Release cycle related (issues) > > The document tries to label all discussion items, such that it is easy to > comment. I normally attach a converted markdown version: however, this is > unwieldly in this case, because there is a large number of tables and > images. Thus, I have created a google doc copy which allows anyone with the > following link > https://docs.google.com/document/d/1FbGV4ZZB9OU8SI4b9ntnM-l6NaQLND8Yfd9u11V > 5Q5A/edit?usp=sharing to comment on sections of the document. If you do, > please make sure you identify yourself in the comment and/or also highlight > feedback in the e-mail thread discussion that will follow this document. > > Please also let us know areas of the whitepaper you agree with, as this will > make it overall easier to identify how much consensus there would be to > address specific issues and proposals in the document. Otherwise the > discussion will primarily focus on points of contention, while other areas > where in fact there may be consensus, will be missed. If there is little or > no feedback (either positive or negative), we have to assume that people > are happy with the status quo and that there is only a weak case for > changes. > Best Regards > Lars > > > signature.asc Description: This is a digitally signed message part. ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] Xen Project Security Whitepaper v1 is ready for community review
On 18/05/2018, 15:07, "Wei Liu" wrote: On Fri, May 18, 2018 at 07:16:24AM -0600, Jan Beulich wrote: > >>> On 18.05.18 at 12:13, wrote: > > 2.2.4.A > > We've discussed the option of different support life times before (and iirc more > than once). Personally I continue to think that all releases should be equal. > > I also continue to think that it was a mistake to shorten the release cadence to > 6 months, for the very reason of there being too many active releases. > We have run this for long enough, we can certainly revisit this topic. If a short release cadence only creates burdens with no visible benefit, we should change it. I suppose we should run a consultation like this one at some point. I already had something for Q3 on my plan, with an intention to have a discussion about it at the developer summit. We can use the outcome as a starting point Regards Lars ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] Xen Project Security Whitepaper v1 is ready for community review
On Fri, May 18, 2018 at 07:16:24AM -0600, Jan Beulich wrote: > >>> On 18.05.18 at 12:13, wrote: > > 2.2.4.A > > We've discussed the option of different support life times before (and iirc > more > than once). Personally I continue to think that all releases should be equal. > > I also continue to think that it was a mistake to shorten the release cadence > to > 6 months, for the very reason of there being too many active releases. > We have run this for long enough, we can certainly revisit this topic. If a short release cadence only creates burdens with no visible benefit, we should change it. I suppose we should run a consultation like this one at some point. Wei. ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] Xen Project Security Whitepaper v1 is ready for community review
> On May 18, 2018, at 06:13, Lars Kurth wrote: > > Dear Community Members, > > just under 3 months ago, we started a community consultation titled "Xen > Security Process Consultation: is there a case to change anything?" (see > https://lists.xenproject.org/archives/html/xen-announce/2018-02/msg0.html). > As promised, I would collate the input - together with further analysis > trying to genuinely consider the implications of what respondents to the > consultation have been suggesting - in a white paper. The white paper is > attached The paper title would be more precise if "Security" was replaced by "Security Process". Rich ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] Xen Project Security Whitepaper v1 is ready for community review
>>> On 18.05.18 at 12:13, wrote: 2.2.4.A We've discussed the option of different support life times before (and iirc more than once). Personally I continue to think that all releases should be equal. I also continue to think that it was a mistake to shorten the release cadence to 6 months, for the very reason of there being too many active releases. 3.1. R1 Naming batching as a possible option in the policy would certainly be nice. I wouldn't want to see it become anywhere close to mandatory though. 3.1. R2 Workload is equally an issue for the security team itself, when the batches grow too large. However, as sometimes reports of several new issues simply happen to occur at close succession. In such a case, I'd rather not see some of the issues artificially delayed, unless they're really minor. And even for really minor ones, allowing for such a delay implies a non-zero risk of delaying for an arbitrarily large amount of time. The conclusion of allowing the security team some room in their decisions without really formalizing anything here sounds fine to me. 3.1. R3 A fixed schedule has some drawbacks which I didn't see mentioned: It creates pressure on the security team to get things ready. At times such pressure is useful (in order to ensure forward progress), but it can also become a problem: We better don't issue half-baked patches, increasing the risk of re-issues or even follow-on XSAs. The alternative risk is that things may get delayed for an overly long period. I don't think it is a secret that at times we as the security team already sit on issues for far too long. 3.2. (2.2.2 A) What is the alternative of re-issues? Not re-issuing, and keeping an issue with the patches or text un-addressed? 3.2. (3.2) I'm unconvinced that R3 really addresses this. Having a fixed date on which a release should occur is quite likely very desirable for PR reasons, but is rather unrealistic. In fact, as expressed before, I'm against any such kind of fixed schedule - a release should be made when it's ready, not when some schedule says there should be a release. There should in particular not be any hindrance to delaying a release for an important bug fix, be it a security one or not. Jan ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel