Re: [RFC PATCH] docker: Add support for using eatmydata in the database
Russell Currey writes: > When running tox on a VM with presumably pretty busy spinning disks, > using eatmydata with the database took running one configuration's test > suite from (no exaggeration) 20 minutes down to 60 seconds. As the author and been-attempting-to-no-longer-be-maintainer-of eatmydata, this 20x improvement in test execution time doesn't really surprise me. It turns out that properly flushing things to disk is *really* expensive. -- Stewart Smith OPAL Architect, IBM. ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: [PATCH] docs: Document backport criteria
Stephen Finucane writes: > Explain why we don't want to be in the business of backport certain > patches, in the long run. It took me a while to put this into words but > I was helped by a similar discussion ongoing in the OpenStack community > at the moment [1]. > > [1] > http://lists.openstack.org/pipermail/openstack-discuss/2019-May/006220.html > > Signed-off-by: Stephen Finucane > Cc: Daniel Axtens > --- > docs/development/releasing.rst | 27 +++ > 1 file changed, 27 insertions(+) > > diff --git a/docs/development/releasing.rst b/docs/development/releasing.rst > index 86cacb3a..8bb6b314 100644 > --- a/docs/development/releasing.rst > +++ b/docs/development/releasing.rst > @@ -115,3 +115,30 @@ when committing:: > > When enough patches have been backported, you should release a new **PATCH** > release. > + > +Backport criteria > +~ > + > +We consider bug fixes and security updates to the Patchwork code itself valid > +for backporting, along with fixes to documentation and developer tooling. We > do > +not, however, consider the following backportable: > + > +Features > + Backporting features is complicated and introduces instability in what is > + supposed to be stable release. If new features are required, users should > + update their Patchwork version. Agreed. > + > +API changes > + Except for bug fixes that resolve 5xx-class errors or fix security issues. > + This also applies to API versions. Agreed. (Unless I've misread it, the first line is incomplete: except for ..., what? I know from context that the 'what' is no backports, but it's not clear from the text.) > + > +Requirement changes > + Requirements on a stable branch are provided as a "snapshot in time" and, > as > + with features, should not change so as to prevent instability being > introduced > + in a stable branch. In addition, stable requirements are not a mechanism to > + alert users to security vulnerabilities and should not be considered as > such. I don't think I really buy the idea that a snapshot in time is a particularly useful or meaningful way of conceptualising an individual software component's release in large and highly interdependent systems. But, I don't think that's worth litigating here in light of the carve-out for distro support below. I am with you on the "In addition" part. > + Users of stable branches should either rely on distro-provided > dependencies, > + which generally maintain a snapshot-in-time fork of packages and > selectively > + backport fixes to them, or manage dependencies manually. In cases, where > using (no comma needed after cases?) > + a distro-provided package necessitates minor changes to the Patchwork code, > + these can be discussed on a case-by-case basis. I think this is generally OK. I would have made a stronger statement about merging small patches to support distro-provided packages if I had written it, but I think the practical upshot of our development process means that discussion (or at least evaluation) of patches on a case-by-case basis is an accurate description of what will inevitably need to occur to get patches in to stable trees. I'd be really wary about suggesting that people manage dependencies themselves, as they then take on the burden of tracking security updates for their systems themselves and this is hard work. One final though I had: when I was working for Canonical in their support engineering team we had rules for what could make it in to stable kernels - https://wiki.ubuntu.com/KernelTeam/KernelUpdates The first four rules are the usual boring sorts of things, critical bug fixes, tracking stable kernel releases, etc. Their final category for patch acceptance, however, reads: "$DEITY intervention. Might happen, but very very rarely and will not be explainable." Do we want a similar thing for if and when we decide to break the rules, or are we right to leave that as implied? Regards, Daniel > -- > 2.21.0 ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: [PATCH] Revert "parser: Ensure whitespace is stripped for long headers"
Stephen Finucane writes: > On Tue, 2019-05-14 at 16:11 +1000, Daniel Axtens wrote: >> This reverts commit 841f966b8d54b2f51ab1c498eed6e5391f2546a9. >> >> In July 2018, we received a report of OzLabs patchwork mangling >> emails that have subjects containing words with internal commas, >> like "Insert DT binding for foo,bar" (#197). >> >> Stephen took a look and came up with the comment this reverts. Quoting >> the commit message: >> >> RFC2822 states that long headers can be wrapped using CRLF followed by >> WSP [1]. For example: >> >> Subject: Foo bar, >> baz >> >> Should be parsed as: >> >> Foo bar,baz >> >> As it turns out, this is not the case. Journey with me to >> section 2.2.3 of RFC 2822: >> >> 2.2.3. Long Header Fields >> >>Each header field is logically a single line of characters comprising >>the field name, the colon, and the field body. For convenience >>however, and to deal with the 998/78 character limitations per line, >>the field body portion of a header field can be split into a multiple >>line representation; this is called "folding". The general rule is >>that wherever this standard allows for folding white space (not >>simply WSP characters), a CRLF may be inserted before any WSP. For >>example, the header field: >> >>Subject: This is a test >> >>can be represented as: >> >>Subject: This >> is a test >> >> So the issue with the example in the reverted commit is that >> there is no folding white space in "bar,baz", so it's not valid >> to split it. >> >> These are valid: >> Subject: Foo bar,baz >> Subject: Foo >> bar,baz >> >> but splitting "bar,baz" into "bar,\n baz" is not valid. >> >> What then is correct unfolding behaviour? Quoting the RFC again: >> >>The process of moving from this folded multiple-line representation >>of a header field to its single line representation is called >>"unfolding". Unfolding is accomplished by simply removing any CRLF >>that is immediately followed by WSP. Each header field should be >>treated in its unfolded form for further syntactic and semantic >>evaluation. >> >> In other words, the unfolding rule requires you to strip the CRLF, >> but it does not permit you to strip the WSP. Indeed, if "bar,\n baz" >> is received, the correct unfolding is "bar, baz". >> >> If you do strip the WSP, you end up mashing words together, such >> as in https://patchwork.ozlabs.org/patch/1097852/ >> >> So revert the commit, restoring original behaviour, but keep a >> corrected version of the test. >> >> This presents a big question though: how did Rob's email up with >> a mangled subject line? >> >> To answer this question, you end up having to learn about OzLabs >> Patchwork and how it differs from Patchwork the project. >> >> OzLabs Patchwork (patchwork.ozlabs.org) is an installation of >> Patchwork. Part of what makes it so useful for so many projects is >> a little intervening layer that can massage some mail to make it >> end up in the right project. Email that lands in the device tree >> project is an example of email that goes through this process. >> I only learned about this today and I haven't looked in any detail >> at precisely what is done to the mail. The script is not part of the >> Patchwork project. >> >> This intervening filter is a Python script that runs - and this >> is an important detail - in Python 2.7. >> >> Ignoring all the details, the filter basically operates in a pipe >> between the mail program and patchwork's parsemail, like >> >> (mail from system) | filter.py | parsemail >> >> At it's very simplest, filter.py acts as follows: >> >> import email >> import sys >> mail = email.parse_from_file(sys.stdin) >> sys.stdout.write(mail.as_string()) >> >> Fascinatingly, if you take Rob's email from #197 and put it through >> this process, you can see that it is getting mangled: >> >> Before: >> Subject: [PATCH v2 3/4] dt-bindings: sound: wm8994: document wlf,csnaddr-pd >> property >> >> After: >> Subject: [PATCH v2 3/4] dt-bindings: sound: wm8994: document wlf, >> csnaddr-pd property >> >> You can see that python27 has incorrectly wrapped the header, breaking >> where there is not a foldable space. Python3 does not have this issue. >> >> To summarise: >> - part of the magic of OzLabs PW is a filter to make sure mail gets >>to the right place. This isn't part of the Patchwork project and >>so is usually invisible to patchwork developers. >> >> - the filter is written in python27. The email module in py27 has >>a bug that incorrectly breaks subjects around commas within words. >> >> - patchwork correctly unfolds those broken subjects with a space >>after the comma. >> >> - thr extra space was interpreted as a bug in patchwork, leading to >>a misinterpretation of the spec to strip out the whitespace that >>was believed to
[PATCH] docs: Document backport criteria
Explain why we don't want to be in the business of backport certain patches, in the long run. It took me a while to put this into words but I was helped by a similar discussion ongoing in the OpenStack community at the moment [1]. [1] http://lists.openstack.org/pipermail/openstack-discuss/2019-May/006220.html Signed-off-by: Stephen Finucane Cc: Daniel Axtens --- docs/development/releasing.rst | 27 +++ 1 file changed, 27 insertions(+) diff --git a/docs/development/releasing.rst b/docs/development/releasing.rst index 86cacb3a..8bb6b314 100644 --- a/docs/development/releasing.rst +++ b/docs/development/releasing.rst @@ -115,3 +115,30 @@ when committing:: When enough patches have been backported, you should release a new **PATCH** release. + +Backport criteria +~ + +We consider bug fixes and security updates to the Patchwork code itself valid +for backporting, along with fixes to documentation and developer tooling. We do +not, however, consider the following backportable: + +Features + Backporting features is complicated and introduces instability in what is + supposed to be stable release. If new features are required, users should + update their Patchwork version. + +API changes + Except for bug fixes that resolve 5xx-class errors or fix security issues. + This also applies to API versions. + +Requirement changes + Requirements on a stable branch are provided as a "snapshot in time" and, as + with features, should not change so as to prevent instability being introduced + in a stable branch. In addition, stable requirements are not a mechanism to + alert users to security vulnerabilities and should not be considered as such. + Users of stable branches should either rely on distro-provided dependencies, + which generally maintain a snapshot-in-time fork of packages and selectively + backport fixes to them, or manage dependencies manually. In cases, where using + a distro-provided package necessitates minor changes to the Patchwork code, + these can be discussed on a case-by-case basis. -- 2.21.0 ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: [PATCH] Revert "parser: Ensure whitespace is stripped for long headers"
On Tue, 2019-05-14 at 16:11 +1000, Daniel Axtens wrote: > This reverts commit 841f966b8d54b2f51ab1c498eed6e5391f2546a9. > > In July 2018, we received a report of OzLabs patchwork mangling > emails that have subjects containing words with internal commas, > like "Insert DT binding for foo,bar" (#197). > > Stephen took a look and came up with the comment this reverts. Quoting > the commit message: > > RFC2822 states that long headers can be wrapped using CRLF followed by > WSP [1]. For example: > > Subject: Foo bar, > baz > > Should be parsed as: > > Foo bar,baz > > As it turns out, this is not the case. Journey with me to > section 2.2.3 of RFC 2822: > > 2.2.3. Long Header Fields > >Each header field is logically a single line of characters comprising >the field name, the colon, and the field body. For convenience >however, and to deal with the 998/78 character limitations per line, >the field body portion of a header field can be split into a multiple >line representation; this is called "folding". The general rule is >that wherever this standard allows for folding white space (not >simply WSP characters), a CRLF may be inserted before any WSP. For >example, the header field: > >Subject: This is a test > >can be represented as: > >Subject: This > is a test > > So the issue with the example in the reverted commit is that > there is no folding white space in "bar,baz", so it's not valid > to split it. > > These are valid: > Subject: Foo bar,baz > Subject: Foo >bar,baz > > but splitting "bar,baz" into "bar,\n baz" is not valid. > > What then is correct unfolding behaviour? Quoting the RFC again: > >The process of moving from this folded multiple-line representation >of a header field to its single line representation is called >"unfolding". Unfolding is accomplished by simply removing any CRLF >that is immediately followed by WSP. Each header field should be >treated in its unfolded form for further syntactic and semantic >evaluation. > > In other words, the unfolding rule requires you to strip the CRLF, > but it does not permit you to strip the WSP. Indeed, if "bar,\n baz" > is received, the correct unfolding is "bar, baz". > > If you do strip the WSP, you end up mashing words together, such > as in https://patchwork.ozlabs.org/patch/1097852/ > > So revert the commit, restoring original behaviour, but keep a > corrected version of the test. > > This presents a big question though: how did Rob's email up with > a mangled subject line? > > To answer this question, you end up having to learn about OzLabs > Patchwork and how it differs from Patchwork the project. > > OzLabs Patchwork (patchwork.ozlabs.org) is an installation of > Patchwork. Part of what makes it so useful for so many projects is > a little intervening layer that can massage some mail to make it > end up in the right project. Email that lands in the device tree > project is an example of email that goes through this process. > I only learned about this today and I haven't looked in any detail > at precisely what is done to the mail. The script is not part of the > Patchwork project. > > This intervening filter is a Python script that runs - and this > is an important detail - in Python 2.7. > > Ignoring all the details, the filter basically operates in a pipe > between the mail program and patchwork's parsemail, like > > (mail from system) | filter.py | parsemail > > At it's very simplest, filter.py acts as follows: > > import email > import sys > mail = email.parse_from_file(sys.stdin) > sys.stdout.write(mail.as_string()) > > Fascinatingly, if you take Rob's email from #197 and put it through > this process, you can see that it is getting mangled: > > Before: > Subject: [PATCH v2 3/4] dt-bindings: sound: wm8994: document wlf,csnaddr-pd > property > > After: > Subject: [PATCH v2 3/4] dt-bindings: sound: wm8994: document wlf, > csnaddr-pd property > > You can see that python27 has incorrectly wrapped the header, breaking > where there is not a foldable space. Python3 does not have this issue. > > To summarise: > - part of the magic of OzLabs PW is a filter to make sure mail gets >to the right place. This isn't part of the Patchwork project and >so is usually invisible to patchwork developers. > > - the filter is written in python27. The email module in py27 has >a bug that incorrectly breaks subjects around commas within words. > > - patchwork correctly unfolds those broken subjects with a space >after the comma. > > - thr extra space was interpreted as a bug in patchwork, leading to >a misinterpretation of the spec to strip out the whitespace that >was believed to be in error. > > - that broke other wrapped subjects. > > To solve this, revert the commit and I'll work with jk to get the > filter script into py3
[PATCH] Revert "parser: Ensure whitespace is stripped for long headers"
This reverts commit 841f966b8d54b2f51ab1c498eed6e5391f2546a9. In July 2018, we received a report of OzLabs patchwork mangling emails that have subjects containing words with internal commas, like "Insert DT binding for foo,bar" (#197). Stephen took a look and came up with the comment this reverts. Quoting the commit message: RFC2822 states that long headers can be wrapped using CRLF followed by WSP [1]. For example: Subject: Foo bar, baz Should be parsed as: Foo bar,baz As it turns out, this is not the case. Journey with me to section 2.2.3 of RFC 2822: 2.2.3. Long Header Fields Each header field is logically a single line of characters comprising the field name, the colon, and the field body. For convenience however, and to deal with the 998/78 character limitations per line, the field body portion of a header field can be split into a multiple line representation; this is called "folding". The general rule is that wherever this standard allows for folding white space (not simply WSP characters), a CRLF may be inserted before any WSP. For example, the header field: Subject: This is a test can be represented as: Subject: This is a test So the issue with the example in the reverted commit is that there is no folding white space in "bar,baz", so it's not valid to split it. These are valid: Subject: Foo bar,baz Subject: Foo bar,baz but splitting "bar,baz" into "bar,\n baz" is not valid. What then is correct unfolding behaviour? Quoting the RFC again: The process of moving from this folded multiple-line representation of a header field to its single line representation is called "unfolding". Unfolding is accomplished by simply removing any CRLF that is immediately followed by WSP. Each header field should be treated in its unfolded form for further syntactic and semantic evaluation. In other words, the unfolding rule requires you to strip the CRLF, but it does not permit you to strip the WSP. Indeed, if "bar,\n baz" is received, the correct unfolding is "bar, baz". If you do strip the WSP, you end up mashing words together, such as in https://patchwork.ozlabs.org/patch/1097852/ So revert the commit, restoring original behaviour, but keep a corrected version of the test. This presents a big question though: how did Rob's email up with a mangled subject line? To answer this question, you end up having to learn about OzLabs Patchwork and how it differs from Patchwork the project. OzLabs Patchwork (patchwork.ozlabs.org) is an installation of Patchwork. Part of what makes it so useful for so many projects is a little intervening layer that can massage some mail to make it end up in the right project. Email that lands in the device tree project is an example of email that goes through this process. I only learned about this today and I haven't looked in any detail at precisely what is done to the mail. The script is not part of the Patchwork project. This intervening filter is a Python script that runs - and this is an important detail - in Python 2.7. Ignoring all the details, the filter basically operates in a pipe between the mail program and patchwork's parsemail, like (mail from system) | filter.py | parsemail At it's very simplest, filter.py acts as follows: import email import sys mail = email.parse_from_file(sys.stdin) sys.stdout.write(mail.as_string()) Fascinatingly, if you take Rob's email from #197 and put it through this process, you can see that it is getting mangled: Before: Subject: [PATCH v2 3/4] dt-bindings: sound: wm8994: document wlf,csnaddr-pd property After: Subject: [PATCH v2 3/4] dt-bindings: sound: wm8994: document wlf, csnaddr-pd property You can see that python27 has incorrectly wrapped the header, breaking where there is not a foldable space. Python3 does not have this issue. To summarise: - part of the magic of OzLabs PW is a filter to make sure mail gets to the right place. This isn't part of the Patchwork project and so is usually invisible to patchwork developers. - the filter is written in python27. The email module in py27 has a bug that incorrectly breaks subjects around commas within words. - patchwork correctly unfolds those broken subjects with a space after the comma. - thr extra space was interpreted as a bug in patchwork, leading to a misinterpretation of the spec to strip out the whitespace that was believed to be in error. - that broke other wrapped subjects. To solve this, revert the commit and I'll work with jk to get the filter script into py3 compatibility. (Given that py27 sunsets in ~7mo, trying to fix it is not worth it.) Closes: #197 CC: stable --- patchwork/parser.py| 1 - patchwork/tests/test_parser.py | 2 +- releasenotes/notes/issue-197-4f7594db1e4c9887.yaml | 9 + 3 files changed, 6