Re: zlib acceleration?
On May 29, 2015, at 8:52 AM, Rainer Müller wrote: > Even if this CloudFlare patch offers any performance improvements, the > authors did not bother to implement it the sane way respecting licenses > for further distribution. In this form, it will never be included > upstream and we should not do it either. +1 > The Intel patch looks better, applying zlib license to the files added. > Also it has a check for availability of the SSE 4.2 feature before > execution of the machine instruction. > > Unfortunately, I couldn't find any zlib mailing list archive containing > the discussion of including the Intel patches. https://web.archive.org/web/20131201003911/http://mail.madler.net/pipermail/zlib-devel_madler.net/2013-November/thread.html vq ___ macports-users mailing list macports-users@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-users
Re: zlib acceleration?
On Friday May 29 2015 14:52:04 Rainer Müller wrote: >Looking at the CloudFlare patch, the implementation does not seem to >check for availability of the SSE 4 instruction set with CPUID and does >not offer any fallback. No, but one could do the test before building, even to the extent of rejecting the variant if the host doesn't support SSE4. Which would be enough if the patched port is never distributed in binary form. >Especially this GPL header here bothers me a lot: >https://github.com/cloudflare/zlib/blob/gcc.amd64/contrib/amd64/crc32-pclmul_asm.S That file doesn't build on OS X so it'd be excluded in any event, so if that's the only file concerned by licensing issues the argument is moot. My plan is to create a patch of CloudFlare's sources against the release stock zlib version, so it'll be easy enough to remove the file before making the patch. >After applying this to the zlib port, the resulting binary would >certainly be covered by the GPL-2 and not the zlib license anymore. That >would introduce lots of compatibility problems. I don't see how it would do more than make the port non-distributable in binary form, which is a moot point if the patch is included in a non-default variant as I plan to. And anyone who distributes software with the binary library included will have the option of installing the port in stock zlib form, regardless of whether the "turbo" variant is default or not. > >Even if this CloudFlare patch offers any performance improvements, the Have you actually looked at the benchmark results? >for further distribution. In this form, it will never be included >upstream and we should not do it either. MacPorts users are grown-up and vaccinated for the most part, and can make licensing related decisions for themselves (and bear any consequences). In any case, I'll do as announced, and post message when I have an adapted port I'm satisfied with. >Unfortunately, I couldn't find any zlib mailing list archive containing >the discussion of including the Intel patches. It's supposed to have taken place, but the only link I've seen has gone stale. R. ___ macports-users mailing list macports-users@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-users
Re: zlib acceleration?
On 2015-05-29 11:21, René J.V. Bertin wrote: > After re-running the benchmarks with the HEADS from the respective > implementations, the picture changes and the CloudFlare patch becomes > much more interesting. The CloudFlare patch in its current form does > require the -msse4 flag as it uses a crc intrinsic from that instruction > set. Are there still Macs around that do *not* have SSE 4.1, supposing > they ever existed (apart from PPC Macs of course)? There are Macs from 2006/2007 with a 32-bit Intel Core Duo processor that do not support SSE 4.1. It is doubtful whether the default compilers on these old platforms support these instruction sets. However, that should not be a problem, as we could probably exclude these old Macs by enabling the feature only on recent enough versions of OS X which require newer processors anyway. Looking at the CloudFlare patch, the implementation does not seem to check for availability of the SSE 4 instruction set with CPUID and does not offer any fallback. However, the rest of the patch does not look applicable either. Especially this GPL header here bothers me a lot: https://github.com/cloudflare/zlib/blob/gcc.amd64/contrib/amd64/crc32-pclmul_asm.S Furthermore, "Function stolen from linux kernel 3.14.", although they did not even add any GPL copyright notice to the file: https://github.com/cloudflare/zlib/blob/gcc.amd64/crc32.c#L244 After applying this to the zlib port, the resulting binary would certainly be covered by the GPL-2 and not the zlib license anymore. That would introduce lots of compatibility problems. Even if this CloudFlare patch offers any performance improvements, the authors did not bother to implement it the sane way respecting licenses for further distribution. In this form, it will never be included upstream and we should not do it either. The Intel patch looks better, applying zlib license to the files added. Also it has a check for availability of the SSE 4.2 feature before execution of the machine instruction. Unfortunately, I couldn't find any zlib mailing list archive containing the discussion of including the Intel patches. Overall I concur with Ryan. Development of an library as mature as zlib is usually conservative and I would let the upstream maintainers decide if they want to include these patches in their next release. Rainer ___ macports-users mailing list macports-users@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-users
Re: zlib acceleration?
After re-running the benchmarks with the HEADS from the respective implementations, the picture changes and the CloudFlare patch becomes much more interesting. The CloudFlare patch in its current form does require the -msse4 flag as it uses a crc intrinsic from that instruction set. Are there still Macs around that do *not* have SSE 4.1, supposing they ever existed (apart from PPC Macs of course)? (Note: there are with my computer "idle" in the sense I wasn't touching it, but I didn't quit my browsers, MUA and other stuff to let the number be a bit more representative) * baseline cloudflare intel * compress executable -1 (x 5) Compression ratio: 0.37 0.37 0.46 Execution time [s]: 0.68 ± 0.03 (100.00%) 0.50 ± 0.01 ( 72.95%) 0.27 ± 0.00 ( 39.84%) compress executable -3 (x 5) Compression ratio: 0.35 0.36 0.36 Execution time [s]: 1.00 ± 0.00 (100.00%) 0.65 ± 0.00 ( 65.00%) 0.72 ± 0.01 ( 71.77%) compress executable -5 (x 5) Compression ratio: 0.33 0.34 0.34 Execution time [s]: 1.58 ± 0.02 (100.00%) 0.94 ± 0.00 ( 59.65%) 0.98 ± 0.01 ( 62.25%) compress executable -7 (x 5) Compression ratio: 0.33 0.34 0.33 Execution time [s]: 3.54 ± 0.03 (100.00%) 1.69 ± 0.01 ( 47.59%) 2.63 ± 0.03 ( 74.25%) compress executable -9 (x 5) Compression ratio: 0.33 0.33 0.33 Execution time [s]: 8.89 ± 0.06 (100.00%) 4.29 ± 0.02 ( 48.26%) 7.30 ± 0.06 ( 82.08%) compress html -1 (x 5) Compression ratio: 0.39 0.37 0.54 Execution time [s]: 0.36 ± 0.00 (100.00%) 0.28 ± 0.01 ( 78.02%) 0.19 ± 0.00 ( 51.98%) compress html -3 (x 5) Compression ratio: 0.36 0.35 0.35 Execution time [s]: 0.59 ± 0.01 (100.00%) 0.37 ± 0.00 ( 62.99%) 0.42 ± 0.00 ( 71.11%) compress html -5 (x 5) Compression ratio: 0.34 0.33 0.33 Execution time [s]: 0.98 ± 0.01 (100.00%) 0.56 ± 0.00 ( 57.47%) 0.57 ± 0.00 ( 58.09%) compress html -7 (x 5) Compression ratio: 0.33 0.33 0.33 Execution time [s]: 1.65 ± 0.01 (100.00%) 0.90 ± 0.01 ( 54.83%) 1.50 ± 0.02 ( 91.24%) compress html -9 (x 5) Compression ratio: 0.33 0.33 0.33 Execution time [s]: 2.57 ± 0.03 (100.00%) 1.63 ± 0.02 ( 63.45%) 2.34 ± 0.03 ( 91.22%) compress jpeg -1 (x 5) Compression ratio: 1.00 1.00 1.05 Execution time [s]: 0.60 ± 0.01 (100.00%) 0.53 ± 0.00 ( 88.87%) 0.26 ± 0.01 ( 43.95%) compress jpeg -3 (x 5) Compression ratio: 1.00 1.00 1.00 Execution time [s]: 0.61 ± 0.01 (100.00%) 0.57 ± 0.01 ( 94.30%) 0.57 ± 0.00 ( 94.16%) compress jpeg -5 (x 5) Compression ratio: 1.00 1.00 1.00 Execution time [s]: 0.64 ± 0.00 (100.00%) 0.57 ± 0.01 ( 88.49%) 0.73 ± 0.00 (113.72%) compress jpeg -7 (x 5) Compression ratio: 1.00 1.00 1.00 Execution time [s]: 0.65 ± 0.01 (100.00%) 0.58 ± 0.00 ( 89.00%) 0.59 ± 0.00 ( 91.46%) compress jpeg -9 (x 5) Compression ratio: 1.00 1.00 1.00 Execution time [s]: 0.66 ± 0.01 (100.00%) 0.59 ± 0.01 ( 89.78%) 0.62 ± 0.01 ( 94.03%) compress pngpixels -1 (x 5) Compression ratio: 0.17 0.17 0.23 Execution time [s]: 0.43 ± 0.01 (100.00%) 0.29 ± 0.00 ( 68.39%) 0.19 ± 0.00 ( 43.86%) compress pngpixels -3 (x 5) Compression ratio: 0.15 0.15 0.16 Execution time [s]: 0.90 ± 0.01 (100.00%) 0.47 ± 0.00 ( 51.61%) 0.46 ± 0.00 ( 51.12%) compress pngpixels -5 (x 5) Compression ratio: 0.14 0.14 0.14 Execution time [s]: 1.29 ± 0.01 (100.00%) 0.69 ± 0.01 ( 53.80%) 0.72 ± 0.01 ( 55.86%) compress pngpixels -7 (x 5) Compression ratio: 0.13 0.13 0.13 Execution time [s]: 3.94 ± 0.03 (100.00%) 2.09 ± 0.02 ( 52.95%) 3.20 ± 0.03 ( 81.24%) compress pngpixels -9 (x 5) Compression ratio: 0.12 0.12 0.12 Execution time [s]: 28.56 ± 0.08 (100.00%) 15.54 ± 0.05 ( 54.40%) 23.42 ± 0.30 ( 81.99%) decompress executable (x 100) Execution time [s]: 5.92 ± 0.06 (100.00%) 5.91 ± 0.02 ( 99.97%) 6.01 ± 0.02 (101.64%) deco
Re: zlib acceleration?
On Thursday May 28 2015 04:43:28 Ryan Schmidt wrote: > I'm not comfortable with making MacPorts users involuntary testers for > unofficial code in a fundamental library like zlib. Not on an obligatory basis, no, not until we can show that there are no drawbacks to it. I plan to add a variant to a personal version of port:zlib, and publish that via my repository. I see a few possible reasons why upstream hasn't included it: - there was an issue with the original patch, which caused them to hold off on incorporating it. That issue has been resolved. - they find the gains not worth the effort, leaving it up to those for whom a 10% speed increase is significant to build their own copy - zlib development has simply stalled for lack of whatever. Two years without even something as fashionable as a security patch feels like a sign on the wall ... BTW, can someone try the benchmark script I linked to on 10.10 and more recent hardware than mine (early 2011 MBP 13")? Building the CloudFare version with -march=native causes clang to fail in the assembly stage, which reeks of a clang bug I might have to report to Apple. R. ___ macports-users mailing list macports-users@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-users
Re: zlib acceleration?
On May 28, 2015, at 3:30 AM, René J.V. Bertin wrote: > On Thursday May 28 2015 00:14:14 Ryan Schmidt wrote: > >> Instead of trying to add a patch for this to MacPorts, I would rather you >> work with the developers of zlib on including the changes into zlib itself > > When the Phoronix article was written, this was being discussed on the zlib > ML and that's about 2 years ago. If no patches were included in that time, > and even a 2nd party felt the need to develop their own, I think it'd be a > waste of time to try that route. That was kind of my point. > Not without some additional evidence that the patch works and has an interest > without drawbacks, which could come from deploying it through MacPorts for a > while. I'm not comfortable with making MacPorts users involuntary testers for unofficial code in a fundamental library like zlib. Maybe you can include it in your own private copy of zlib, and based on your experiences running that, you can make recommendations to the developers. But if the developers don't believe it should be included, then I don't feel we should second-guess them. ___ macports-users mailing list macports-users@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-users
Re: zlib acceleration?
On Thursday May 28 2015 00:14:14 Ryan Schmidt wrote: >Instead of trying to add a patch for this to MacPorts, I would rather you work >with the developers of zlib on including the changes into zlib itself When the Phoronix article was written, this was being discussed on the zlib ML and that's about 2 years ago. If no patches were included in that time, and even a 2nd party felt the need to develop their own, I think it'd be a waste of time to try that route. Not without some additional evidence that the patch works and has an interest without drawbacks, which could come from deploying it through MacPorts for a while. R. ___ macports-users mailing list macports-users@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-users
Re: zlib acceleration?
On May 27, 2015, at 8:56 PM, René J.V. Bertin wrote: > On Wednesday May 27 2015 17:19:01 Fernando Rodriguez wrote: >> On Wednesday, May 27, 2015 09:50:47 PM René J.V. Bertin wrote: >>> Hi, >>> >>> I just came across this: >>> http://www.phoronix.com/scan.php?page=news_item&px=MTUyNzY >>> >>> The link to the zlib ML on that page has gone stale, so I can't check >>> whether those Intel patches ever made it into zlib >> >> The ChangeLog makes no mention of it so I guess not. > > Intel hasn't been the only one working on this: > > https://www.snellman.net/blog/archive/2014-08-04-comparison-of-intel-and-cloudflare-zlib-patches.html > https://github.com/cloudflare/zlib/tree/gcc.amd64 > > I'll run the benchmark from the snellman site, and then see if it's > worthwhile to come up with a patch. Instead of trying to add a patch for this to MacPorts, I would rather you work with the developers of zlib on including the changes into zlib itself, and then getting the developers of zlib to release a new version. Then we can update MacPorts zlib to that version. I don't feel it's MacPorts' responsibility to decide what new features should go into any given package; that's for the developers of that software to decide. ___ macports-users mailing list macports-users@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-users
Re: zlib acceleration?
On Wednesday May 27 2015 17:19:01 Fernando Rodriguez wrote: > On Wednesday, May 27, 2015 09:50:47 PM René J.V. Bertin wrote: > > Hi, > > > > I just came across this: > > http://www.phoronix.com/scan.php?page=news_item&px=MTUyNzY > > > > The link to the zlib ML on that page has gone stale, so I can't check > > whether those Intel patches ever made it into zlib > > The ChangeLog makes no mention of it so I guess not. Intel hasn't been the only one working on this: https://www.snellman.net/blog/archive/2014-08-04-comparison-of-intel-and-cloudflare-zlib-patches.html https://github.com/cloudflare/zlib/tree/gcc.amd64 I'll run the benchmark from the snellman site, and then see if it's worthwhile to come up with a patch. > I don't think bsdtar does, at least it won't use lzip from macports. I just bsdtar from port:libarchive certainly does use the lzip from macports ... R ___ macports-users mailing list macports-users@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-users
Re: zlib acceleration?
On Wednesday, May 27, 2015 09:50:47 PM René J.V. Bertin wrote: > Hi, > > I just came across this: > http://www.phoronix.com/scan.php?page=news_item&px=MTUyNzY > > The link to the zlib ML on that page has gone stale, so I can't check > whether those Intel patches ever made it into zlib The ChangeLog makes no mention of it so I guess not. > zlib is used by afsctool and presumably also by bsdtar for the > --hfsCompression feature, so any speed-up of zlib compression should reduce > compression times in those applications (and presumably a lot of others). I don't think bsdtar does, at least it won't use lzip from macports. I just had to install gnutar to get it to work. ___ macports-users mailing list macports-users@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-users
Re: zlib acceleration?
On May 27, 2015, at 3:50 PM, René J.V. Bertin wrote: > The link to the zlib ML on that page has gone stale, so I can't check whether > those Intel patches ever made it into zlib Probably not, given that the latest upstream release occurred months prior. > Does anyone know what the status is on this? According to livecheck, > port:zlib is up to date ... It *is* up to date. zlib @1.2.8_0 provides the latest upstream release. http://www.zlib.net vq ___ macports-users mailing list macports-users@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-users
zlib acceleration?
Hi, I just came across this: http://www.phoronix.com/scan.php?page=news_item&px=MTUyNzY The link to the zlib ML on that page has gone stale, so I can't check whether those Intel patches ever made it into zlib, but given the date in the zlib Portfile it seems unlikely that the version we have in MacPorts has them. zlib is used by afsctool and presumably also by bsdtar for the --hfsCompression feature, so any speed-up of zlib compression should reduce compression times in those applications (and presumably a lot of others). Does anyone know what the status is on this? According to livecheck, port:zlib is up to date ... R. ___ macports-users mailing list macports-users@lists.macosforge.org https://lists.macosforge.org/mailman/listinfo/macports-users
