Re: FYI brotli
Just a FYI that it looks like a 0.6.x version of the lib will be released v. soon with all that is needed for the module to work and compile as-is... 1.0.0 will be released a little bit later which will simply deprecate/remove the OLD API, which we don't use anyway... That is, the lib change from 0.6.x to 1.0.0 will require no code change at all.
Re: FYI brotli
Funny you mention it. Nginx had it first anyways, and was (perhaps still is) using the deprecated API that dies with libbrotli rev 1.0.0 - part of that delay might have been affording ngnix a chance to adapt. Versioning their installed library should allow both to be installed at once. So... providing our users enough info to actually use the module is worth hardly a nickle? 95c to win an undocumented feature on a bullet list? Providing no documentation to enable it isn't a failure by httpd? Whatever indeed. gzip and brotli, head to head at compression level 5, % cpu load increase is generally larger than the % bytes saved. brotli becomes much more interesting at much higher pre-compression values, with a corresponding savings in cpu due to fewer bytes read/transmitted. On Feb 16, 2017 13:56, "Jim Jagielski" wrote: Whatever... nginx will have it 1st anyway. And once again we fail our users by having a nickel holding up a dollar. > On Feb 16, 2017, at 2:48 PM, William A Rowe Jr wrote: > > On Thu, Feb 16, 2017 at 12:47 PM, Jim Jagielski wrote: >> >>> On Feb 16, 2017, at 1:15 PM, William A Rowe Jr wrote: >>> >>> >>> I concur with Evgeny Kotkov that an ABI stable dependency is appropriate >>> before adding this to httpd 2.4.x - so far as I've read none have suggested >>> this as an experimental addition to 2.4. >> >> I do. We release it w/ the caveat that it is dependent on a >> library that may change. Until that happens, we have users able >> to use Brotli. That is a Good Thing IMO. > > Let's examine your claim. There is no documentation at all of where to > find brotli. http://brotli.org/ is now one day old (congrats!!!) - but not yet > exactly discoverable, http://lmgtfy.com/?q=brotli+package > > The package lives now in Debian stretch and sid. Archlinux packages > seem out-of-sync. This only exists on Fedora as copr package so far. > So without telling users where to find it, "users able to use Brotli" is a > false statement. > > OS distributors are waiting for 1.0.0 to package this. What's our rush? > >> Re: the docs, seem a minor nit to hold back, esp when you >> don't even quantify how the docs are a complete mess nor the >> requirement for an example in docs/conf. What other requirements >> will you be spinning up? > > Missing 1. where to find brotli? 2. working config examples? These > points might seem counter-intuitive, but without such simple things, > the code is just dead weight. > > I'm investigating the interaction between brotli and deflate so we can > support both simultaniously before feeding it to our stable users. Halting > compression for half of the browsers in order to win a small percentage > gain for the other half of the browsers seems sub-optimal, no? > > I'll be glad to discover that the brotli and deflate filters already coexist > peacefully without double-compressing.
Re: FYI brotli
On Mon, Jan 16, 2017 at 2:28 PM, Evgeny Kotkov wrote: > > There is, however, a potential problem with backporting mod_brotli, since > it relies on the Brotli library 1.0.0, which has not yet been released. > In other words, if the upstream changes the API or the library layout > or their pkg-config files after mod_brotli is shipped with httpd 2.4.x, it's > either going to stop building or working. ... or stop being updated, if the 0.5.x branch is abandoned and the new library versioned for binary compatibility. That said, isn't it more effective for brotli to reversion the lib*.so.n.n.n files to facilitate users of the already compiled ngnix module? The entire point of versioning is to allow multiple consumers to use one of several binary builds. > My impression on this is that mod_brotli is only safe to backport after the > Brotli authors publish the 1.0.0 version of their library. But perhaps I am > missing something? > > (Apart from this, I think that Brotli did change the layout of their > pkg-config > files in [https://github.com/google/brotli/commit/fe9f9a91], and it requires > an update in the filters/config.m4 file; I'll do that.) They may change the package composition again, IIUC, renaming /usr/bin/bro to /usr/bin/brotli or similar, due to an established /usr/bin/bro in many linux distributors' packages. But a build from 1.0.0 trunk today indicates this hasn't happened yet. Thanks again for your earlier clarifications.
Re: FYI brotli
On Thu, Feb 16, 2017 at 2:27 PM, Evgeny Kotkov wrote: > William A Rowe Jr writes: > >> My open questions; has this been entirely reviewed in conjunction with h2? >> Will A-E: br,gzip,deflate axe all others from that list when deciding to >> enable brotli? (I presume not-yet.) Will gzip filter work where A-E: gzip was >> given without br, or are we ceasing to encode half of the web if the user >> elects to serve brotli compression? > > Hi William, > > Answering some of the questions: > > 1) I did test mod_brotli in conjunction with mod_http2 around the time it >was committed. As far as I remember, I didn't spot anything unusual >or any issues. Cool; would be nice to do something in the perl-framework. > 2) The brotli and deflate output filters can coexist. > >Moreover, mod_brotli was written with a particular use case in mind >where this module is added to an existing mod_deflate installation, >and results in sending brotli-encoded data only to the clients that >advertise they know how to deal with it via "Accept-Encoding: br". > >"Accept-Encoding: br, gzip, deflate" is not going to double-encode >the data, as both mod_brotli and mod_deflate are smart enough to only >kick in for identity Content-Encoding. "Accept-Encoding: gzip" is going >to use only the mod_deflate's filter, and mod_brotli will remove itself >from the chain, after not finding "br" in the Accept-Encoding. > >There are a couple of tests from https://svn.apache.org/r1761716 that >verify this behavior. (However, I think that right now there is no >explicit test for the case of sending "Accept-Encoding: gzip, deflate" >to a location with both mod_brotli and mod_deflate.) Indeed - I just expanded it to test this combination and the test passes. Very reassuring. I don't think we need to consider the priority choice of the user-agent, it's on our cpu-intensive server to elect it's idea of best compression. We could add a test for inverted filter order but I don't consider that particularly critical.
Re: FYI brotli
William A Rowe Jr writes: > My open questions; has this been entirely reviewed in conjunction with h2? > Will A-E: br,gzip,deflate axe all others from that list when deciding to > enable brotli? (I presume not-yet.) Will gzip filter work where A-E: gzip was > given without br, or are we ceasing to encode half of the web if the user > elects to serve brotli compression? Hi William, Answering some of the questions: 1) I did test mod_brotli in conjunction with mod_http2 around the time it was committed. As far as I remember, I didn't spot anything unusual or any issues. 2) The brotli and deflate output filters can coexist. Moreover, mod_brotli was written with a particular use case in mind where this module is added to an existing mod_deflate installation, and results in sending brotli-encoded data only to the clients that advertise they know how to deal with it via "Accept-Encoding: br". "Accept-Encoding: br, gzip, deflate" is not going to double-encode the data, as both mod_brotli and mod_deflate are smart enough to only kick in for identity Content-Encoding. "Accept-Encoding: gzip" is going to use only the mod_deflate's filter, and mod_brotli will remove itself from the chain, after not finding "br" in the Accept-Encoding. There are a couple of tests from https://svn.apache.org/r1761716 that verify this behavior. (However, I think that right now there is no explicit test for the case of sending "Accept-Encoding: gzip, deflate" to a location with both mod_brotli and mod_deflate.) Regards, Evgeny Kotkov
Re: FYI brotli
Whatever... nginx will have it 1st anyway. And once again we fail our users by having a nickel holding up a dollar. > On Feb 16, 2017, at 2:48 PM, William A Rowe Jr wrote: > > On Thu, Feb 16, 2017 at 12:47 PM, Jim Jagielski wrote: >> >>> On Feb 16, 2017, at 1:15 PM, William A Rowe Jr wrote: >>> >>> >>> I concur with Evgeny Kotkov that an ABI stable dependency is appropriate >>> before adding this to httpd 2.4.x - so far as I've read none have suggested >>> this as an experimental addition to 2.4. >> >> I do. We release it w/ the caveat that it is dependent on a >> library that may change. Until that happens, we have users able >> to use Brotli. That is a Good Thing IMO. > > Let's examine your claim. There is no documentation at all of where to > find brotli. http://brotli.org/ is now one day old (congrats!!!) - but not yet > exactly discoverable, http://lmgtfy.com/?q=brotli+package > > The package lives now in Debian stretch and sid. Archlinux packages > seem out-of-sync. This only exists on Fedora as copr package so far. > So without telling users where to find it, "users able to use Brotli" is a > false statement. > > OS distributors are waiting for 1.0.0 to package this. What's our rush? > >> Re: the docs, seem a minor nit to hold back, esp when you >> don't even quantify how the docs are a complete mess nor the >> requirement for an example in docs/conf. What other requirements >> will you be spinning up? > > Missing 1. where to find brotli? 2. working config examples? These > points might seem counter-intuitive, but without such simple things, > the code is just dead weight. > > I'm investigating the interaction between brotli and deflate so we can > support both simultaniously before feeding it to our stable users. Halting > compression for half of the browsers in order to win a small percentage > gain for the other half of the browsers seems sub-optimal, no? > > I'll be glad to discover that the brotli and deflate filters already coexist > peacefully without double-compressing.
Re: FYI brotli
On Thu, Feb 16, 2017 at 12:47 PM, Jim Jagielski wrote: > >> On Feb 16, 2017, at 1:15 PM, William A Rowe Jr wrote: >> >> >> I concur with Evgeny Kotkov that an ABI stable dependency is appropriate >> before adding this to httpd 2.4.x - so far as I've read none have suggested >> this as an experimental addition to 2.4. > > I do. We release it w/ the caveat that it is dependent on a > library that may change. Until that happens, we have users able > to use Brotli. That is a Good Thing IMO. Let's examine your claim. There is no documentation at all of where to find brotli. http://brotli.org/ is now one day old (congrats!!!) - but not yet exactly discoverable, http://lmgtfy.com/?q=brotli+package The package lives now in Debian stretch and sid. Archlinux packages seem out-of-sync. This only exists on Fedora as copr package so far. So without telling users where to find it, "users able to use Brotli" is a false statement. OS distributors are waiting for 1.0.0 to package this. What's our rush? > Re: the docs, seem a minor nit to hold back, esp when you > don't even quantify how the docs are a complete mess nor the > requirement for an example in docs/conf. What other requirements > will you be spinning up? Missing 1. where to find brotli? 2. working config examples? These points might seem counter-intuitive, but without such simple things, the code is just dead weight. I'm investigating the interaction between brotli and deflate so we can support both simultaniously before feeding it to our stable users. Halting compression for half of the browsers in order to win a small percentage gain for the other half of the browsers seems sub-optimal, no? I'll be glad to discover that the brotli and deflate filters already coexist peacefully without double-compressing.
Re: FYI brotli
> On Feb 16, 2017, at 1:15 PM, William A Rowe Jr wrote: > > > I concur with Evgeny Kotkov that an ABI stable dependency is appropriate > before adding this to httpd 2.4.x - so far as I've read none have suggested > this as an experimental addition to 2.4. > I do. We release it w/ the caveat that it is dependent on a library that may change. Until that happens, we have users able to use Brotli. That is a Good Thing IMO. Re: the docs, seem a minor nit to hold back, esp when you don't even quantify how the docs are a complete mess nor the requirement for an example in docs/conf. What other requirements will you be spinning up?
Re: FYI brotli
To close up some loose ends/confusion; On Mon, Jan 16, 2017 at 6:42 PM, Jacob Champion wrote: > On 01/16/2017 04:06 PM, William A Rowe Jr wrote: >> >> Before we push this at users.. is there a concern that brotoli >> compression has similar dictionary or simply size based vulnerabilities >> as deflate? > > If you mean HTTP compression oracles (BREACH et al), then I would expect > *any* compression technique to be vulnerable, brotli included. Looking deeper into this class of vulnerability, one of two things must be true; the request headers are encoded (al la TLS compression, which has been dropped from the TLS 1.3 spec), *or* the response body is dynamic and can be abused to decipher the response contents. So there is no issue with serving pre-compressed content, ever (since that is - by definition - static :) There is no issue compressing on the fly any static content. There will be significant risk to some dynamic responses on some sites. >> If so, maybe we teach both to step out of the way when SSL encryption >> filters are in place? > > Current guidance to avoid BREACH is still, AFAIK, to avoid situations where > third-party data is being sent in the same response as first-party secrets. > I don't think we have a way to know when this is happening; it's up to the > server admin to disable compression in dangerous situations. (If anyone > knows of an update to this, please jump in.) > > Disabling compression whenever TLS is enabled (assuming I've understood your > suggestion correctly) is IMO too broad a stroke, and will de-optimize sites > that have been correctly designed to avoid compression oracle attacks. Agreed. Note that Google decided against using T-E: br for any http:// traffic due to frequent proxy misconfiguration. /shrug - Firefox is willing and we have no reason to explicitly disable it. I concur with Evgeny Kotkov that an ABI stable dependency is appropriate before adding this to httpd 2.4.x - so far as I've read none have suggested this as an experimental addition to 2.4. Between this and the lack of any example in docs/conf/ and a complete mess of an example here it's obvious that this proposal is premature; http://httpd.apache.org/docs/trunk/mod/mod_brotli.html#precompressed (Looks like copy-paste errors). My open questions; has this been entirely reviewed in conjunction with h2? Will A-E: br,gzip,deflate axe all others from that list when deciding to enable brotli? (I presume not-yet.) Will gzip filter work where A-E: gzip was given without br, or are we ceasing to encode half of the web if the user elects to serve brotli compression? Looking at that precompression example, it seems like mod_brotli can be patched to avoid double-brotli if the user elected to treat static .br files as either a content-type or a transfer-encoding. We need some way for mod_deflate and mod_brotli to avoid stepping on each other's toes, the toes of any third-party encoding module, and on the administrator's explicit configuration, IMO, before this would be ready for a stable branch.
Re: FYI brotli
Besides, we had no problems supporting OpenSSL 0.9.6 for years :) If/when brotli 1.0.0 is released, we simply add support for that as well. No biggie. > On Jan 17, 2017, at 8:27 AM, Jim Jagielski wrote: > > Actually, it works fine w/ Brotli 0.5.2 which is > what I have installed. > >> On Jan 16, 2017, at 3:28 PM, Evgeny Kotkov >> wrote: >> >> Jim Jagielski writes: >> >>> Functional patch avail... working on doccos. >>> >>> http://home.apache.org/~jim/patches/brotli-2.4.patch >> >> Hi Jim, >> >> Thank you for the backport patch. >> >> There is, however, a potential problem with backporting mod_brotli, since >> it relies on the Brotli library 1.0.0, which has not yet been released. >> In other words, if the upstream changes the API or the library layout >> or their pkg-config files after mod_brotli is shipped with httpd 2.4.x, it's >> either going to stop building or working. >> >> The open ticket about the new release is here: >> >> https://github.com/google/brotli/issues/483 >> >> My impression on this is that mod_brotli is only safe to backport after the >> Brotli authors publish the 1.0.0 version of their library. But perhaps I am >> missing something? >> >> (Apart from this, I think that Brotli did change the layout of their >> pkg-config >> files in [https://github.com/google/brotli/commit/fe9f9a91], and it requires >> an update in the filters/config.m4 file; I'll do that.) >> >> >> Regards, >> Evgeny Kotkov >
Re: FYI brotli
Actually, it works fine w/ Brotli 0.5.2 which is what I have installed. > On Jan 16, 2017, at 3:28 PM, Evgeny Kotkov > wrote: > > Jim Jagielski writes: > >> Functional patch avail... working on doccos. >> >>http://home.apache.org/~jim/patches/brotli-2.4.patch > > Hi Jim, > > Thank you for the backport patch. > > There is, however, a potential problem with backporting mod_brotli, since > it relies on the Brotli library 1.0.0, which has not yet been released. > In other words, if the upstream changes the API or the library layout > or their pkg-config files after mod_brotli is shipped with httpd 2.4.x, it's > either going to stop building or working. > > The open ticket about the new release is here: > > https://github.com/google/brotli/issues/483 > > My impression on this is that mod_brotli is only safe to backport after the > Brotli authors publish the 1.0.0 version of their library. But perhaps I am > missing something? > > (Apart from this, I think that Brotli did change the layout of their > pkg-config > files in [https://github.com/google/brotli/commit/fe9f9a91], and it requires > an update in the filters/config.m4 file; I'll do that.) > > > Regards, > Evgeny Kotkov
Re: FYI brotli
On Mon, 16 Jan 2017 18:06:40 -0600 William A Rowe Jr wrote: > If so, maybe we teach both to step out of the way when SSL encryption > filters are in place? This would make no sense. Brotli is only supported over HTTPS by browsers. Compression-based attacks are a tricky problem, however someone has yet to show that they are abused in practice. But preventing deployment of a new compression algorithm doesn't help. You'd have to disable compression altogether to avoid them. -- Hanno Böck https://hboeck.de/ mail/jabber: ha...@hboeck.de GPG: FE73757FA60E4E21B937579FA5880072BBB51E42
Re: FYI brotli
On 01/16/2017 04:42 PM, Jacob Champion wrote: Current guidance to avoid BREACH is still, AFAIK, to avoid situations where third-party data is being sent in the same response as first-party secrets. I don't think we have a way to know when this is happening ...though if the current response is coming from a static file on disk, that's probably a decent heuristic for the vast majority of users out there... --Jacob
Re: FYI brotli
On 01/16/2017 04:06 PM, William A Rowe Jr wrote: Before we push this at users.. is there a concern that brotoli compression has similar dictionary or simply size based vulnerabilities as deflate? If you mean HTTP compression oracles (BREACH et al), then I would expect *any* compression technique to be vulnerable, brotli included. If so, maybe we teach both to step out of the way when SSL encryption filters are in place? Current guidance to avoid BREACH is still, AFAIK, to avoid situations where third-party data is being sent in the same response as first-party secrets. I don't think we have a way to know when this is happening; it's up to the server admin to disable compression in dangerous situations. (If anyone knows of an update to this, please jump in.) Disabling compression whenever TLS is enabled (assuming I've understood your suggestion correctly) is IMO too broad a stroke, and will de-optimize sites that have been correctly designed to avoid compression oracle attacks. --Jacob
Re: FYI brotli
Before we push this at users.. is there a concern that brotoli compression has similar dictionary or simply size based vulnerabilities as deflate? If so, maybe we teach both to step out of the way when SSL encryption filters are in place? On Jan 16, 2017 10:14, "Jim Jagielski" wrote: > Just a head's up that I am working on the backport proposal/patch > for brotli for 2.4.x... >
Re: FYI brotli
Jim Jagielski writes: > Functional patch avail... working on doccos. > > http://home.apache.org/~jim/patches/brotli-2.4.patch Hi Jim, Thank you for the backport patch. There is, however, a potential problem with backporting mod_brotli, since it relies on the Brotli library 1.0.0, which has not yet been released. In other words, if the upstream changes the API or the library layout or their pkg-config files after mod_brotli is shipped with httpd 2.4.x, it's either going to stop building or working. The open ticket about the new release is here: https://github.com/google/brotli/issues/483 My impression on this is that mod_brotli is only safe to backport after the Brotli authors publish the 1.0.0 version of their library. But perhaps I am missing something? (Apart from this, I think that Brotli did change the layout of their pkg-config files in [https://github.com/google/brotli/commit/fe9f9a91], and it requires an update in the filters/config.m4 file; I'll do that.) Regards, Evgeny Kotkov
Re: FYI brotli
Functional patch avail... working on doccos. http://home.apache.org/~jim/patches/brotli-2.4.patch > On Jan 16, 2017, at 11:11 AM, Jim Jagielski wrote: > > Just a head's up that I am working on the backport proposal/patch > for brotli for 2.4.x...
FYI brotli
Just a head's up that I am working on the backport proposal/patch for brotli for 2.4.x...