Re: svn commit: r278433 - in head: . contrib/xz contrib/xz/src/common contrib/xz/src/liblzma contrib/xz/src/liblzma/api contrib/xz/src/liblzma/api/lzma contrib/xz/src/liblzma/check contrib/xz/src/libl
On Feb 9, 2015, at 05:08, Dmitry Morozovsky ma...@rinet.ru wrote: BTW, Rui, did you some comparative tests with pxz? No, I didn't, but now it should be easy for you to try! :-) -- Rui Paulo ___ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org
Re: svn commit: r278433 - in head: . contrib/xz contrib/xz/src/common contrib/xz/src/liblzma contrib/xz/src/liblzma/api contrib/xz/src/liblzma/api/lzma contrib/xz/src/liblzma/check contrib/xz/src/libl
On Feb 9, 2015, at 08:16, John Baldwin j...@freebsd.org wrote: On Monday, February 09, 2015 06:20:35 AM Rui Paulo wrote: Author: rpaulo Date: Mon Feb 9 06:20:34 2015 New Revision: 278433 URL: https://svnweb.freebsd.org/changeset/base/278433 Log: Merge xz 5.2.0. This brings support for multi-threaded compression. This brings close N times faster compression where N is the number of CPU cores. Because of this, liblzma now depends on libthr. Soon libarchive will be modified to use the new lzma API. Thanks to antoine@ for the exp-run. It looks like this uses hw.ncpu to determine the number of cores. Can you make this cpuset friendly by querying the current process's CPU mask and counting the cores it contains instead? Somethign like: cpuset_t set; cpuset_getaffinity(CPU_LEVEL_WHICH, CPU_WHICH_PID, -1, set, sizeof(set)); (I have a CPU_COUNT() macro I should commit that will then let you do:) ncpus = CPU_COUNT(set); Yes, makes sense. I'll wait for CPU_COUNT(). -- Rui Paulo ___ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org
Re: svn commit: r278433 - in head: . contrib/xz contrib/xz/src/common contrib/xz/src/liblzma contrib/xz/src/liblzma/api contrib/xz/src/liblzma/api/lzma contrib/xz/src/liblzma/check contrib/xz/src/libl
On Monday, February 09, 2015 06:20:35 AM Rui Paulo wrote: Author: rpaulo Date: Mon Feb 9 06:20:34 2015 New Revision: 278433 URL: https://svnweb.freebsd.org/changeset/base/278433 Log: Merge xz 5.2.0. This brings support for multi-threaded compression. This brings close N times faster compression where N is the number of CPU cores. Because of this, liblzma now depends on libthr. Soon libarchive will be modified to use the new lzma API. Thanks to antoine@ for the exp-run. It looks like this uses hw.ncpu to determine the number of cores. Can you make this cpuset friendly by querying the current process's CPU mask and counting the cores it contains instead? Somethign like: cpuset_t set; cpuset_getaffinity(CPU_LEVEL_WHICH, CPU_WHICH_PID, -1, set, sizeof(set)); (I have a CPU_COUNT() macro I should commit that will then let you do:) ncpus = CPU_COUNT(set); -- John Baldwin ___ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org
Re: svn commit: r278433 - in head: . contrib/xz contrib/xz/src/common contrib/xz/src/liblzma contrib/xz/src/liblzma/api contrib/xz/src/liblzma/api/lzma contrib/xz/src/liblzma/check contrib/xz/src/libl
On Mon, Feb 09, 2015 at 04:08:00PM +0300, Dmitry Morozovsky wrote: FWIW, compressing VM images (some sparse files, some not) would take upwards of 45 minutes, which after this update, just takes a few minutes. root@releng2:/R2/vmimages # time xz -T 0 -k FreeBSD-11.0-CURRENT-amd64.qcow2 \ time xz -T 0 -k FreeBSD-11.0-CURRENT-amd64.raw; \ time xz -T 0 -k FreeBSD-11.0-CURRENT-amd64.vhd; \ time xz -T 0 -k FreeBSD-11.0-CURRENT-amd64.vmdk 1027.602u 40.376s 1:09.57 1535.1% 81+192k 0+19774io 0pf+0w 1032.978u 38.823s 1:08.17 1572.2% 81+192k 0+19696io 0pf+0w 1033.908u 38.593s 1:11.70 1495.8% 81+192k 0+19729io 0pf+0w 1091.749u 42.371s 1:04.27 1764.6% 81+192k 0+19751io 0pf+0w I meant to include that this is on a 48-core machine. Hm, I can't beleive you didn't use pxz ;) For RE purposes, using base system utilities supersedes utilities available elsewhere. In my initial tests with pxz, there was an, albeit somewhat predictable, increase in resulting file size as the number of threads increased, while xz in base with the latest update produces output files within +/-1024Kb difference of the unthreaded version. For RE side, there was no real gain in using pxz over xz, because the sacrifice was the output file size. I do not care so much about the time taken to compress the files. I *do* care about the resulting file size, since I (personally) want to be sure that the end user can download the smallest possible file. I was unsure what to expect with the xz(1) update in this regard, and was surprised to see a non-visible difference in the resulting file. Anyway, having this in base, and not depending on external tool, is amazingly great. BTW, Rui, did you some comparative tests with pxz? As stated above, pxz (last I tested) produces incrementally larger files as the thread count increases. From what I have seen so far, the latest xz update does not. Glen pgpU5mCMm0vCx.pgp Description: PGP signature
Re: svn commit: r278433 - in head: . contrib/xz contrib/xz/src/common contrib/xz/src/liblzma contrib/xz/src/liblzma/api contrib/xz/src/liblzma/api/lzma contrib/xz/src/liblzma/check contrib/xz/src/libl
On Feb 8, 2015, at 22:20, Rui Paulo rpa...@freebsd.org wrote: Author: rpaulo Date: Mon Feb 9 06:20:34 2015 New Revision: 278433 URL: https://svnweb.freebsd.org/changeset/base/278433 Log: Merge xz 5.2.0. This brings support for multi-threaded compression. This brings close N times faster compression where N is the number of CPU cores. Because of this, liblzma now depends on libthr. For those that want to try it, this patch adds support to multi-threaded compression when using lzma in libarchive: https://people.freebsd.org/~rpaulo/libarchive-lzma-mt.diff It uses all the available CPU cores by default. -- Rui Paulo ___ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org
Re: svn commit: r278433 - in head: . contrib/xz contrib/xz/src/common contrib/xz/src/liblzma contrib/xz/src/liblzma/api contrib/xz/src/liblzma/api/lzma contrib/xz/src/liblzma/check contrib/xz/src/libl
Rui, On Mon, 9 Feb 2015, Rui Paulo wrote: No, I didn't, but now it should be easy for you to try! :-) Unfortunately, the most powerful machine @myhands is 2*8core root@briareus:~# sysctl hw.model hw.model: Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz root@briareus:~# sysctl hw.ncpu hw.ncpu: 16 and even this one is mostly loaded with $JOB ;) I'll try comparing with my home fileserver, which is on marck@hamster:~ sysctl hw.model hw.model: Intel(R) Core(TM) i5-3570 CPU @ 3.40GHz marck@hamster:~ sysctl hw.ncpu hw.ncpu: 4 and load is rather moderate, especially if I stop smbd ;) -- Sincerely, D.Marck [DM5020, MCK-RIPE, DM3-RIPN] [ FreeBSD committer: ma...@freebsd.org ] *** Dmitry Morozovsky --- D.Marck --- Wild Woozle --- ma...@rinet.ru *** ___ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org
Re: svn commit: r278433 - in head: . contrib/xz contrib/xz/src/common contrib/xz/src/liblzma contrib/xz/src/liblzma/api contrib/xz/src/liblzma/api/lzma contrib/xz/src/liblzma/check contrib/xz/src/libl
On Mon, Feb 09, 2015 at 06:20:35AM +, Rui Paulo wrote: Author: rpaulo Date: Mon Feb 9 06:20:34 2015 New Revision: 278433 URL: https://svnweb.freebsd.org/changeset/base/278433 Log: Merge xz 5.2.0. This brings support for multi-threaded compression. This brings close N times faster compression where N is the number of CPU cores. Because of this, liblzma now depends on libthr. This is fantastic, thank you for doing this update. FWIW, compressing VM images (some sparse files, some not) would take upwards of 45 minutes, which after this update, just takes a few minutes. root@releng2:/R2/vmimages # time xz -T 0 -k FreeBSD-11.0-CURRENT-amd64.qcow2 \ time xz -T 0 -k FreeBSD-11.0-CURRENT-amd64.raw; \ time xz -T 0 -k FreeBSD-11.0-CURRENT-amd64.vhd; \ time xz -T 0 -k FreeBSD-11.0-CURRENT-amd64.vmdk 1027.602u 40.376s 1:09.57 1535.1% 81+192k 0+19774io 0pf+0w 1032.978u 38.823s 1:08.17 1572.2% 81+192k 0+19696io 0pf+0w 1033.908u 38.593s 1:11.70 1495.8% 81+192k 0+19729io 0pf+0w 1091.749u 42.371s 1:04.27 1764.6% 81+192k 0+19751io 0pf+0w Glen pgphsbR0aWmJv.pgp Description: PGP signature
Re: svn commit: r278433 - in head: . contrib/xz contrib/xz/src/common contrib/xz/src/liblzma contrib/xz/src/liblzma/api contrib/xz/src/liblzma/api/lzma contrib/xz/src/liblzma/check contrib/xz/src/libl
Glen, On Mon, 9 Feb 2015, Glen Barber wrote: Log: Merge xz 5.2.0. This brings support for multi-threaded compression. This brings close N times faster compression where N is the number of CPU cores. Because of this, liblzma now depends on libthr. This is fantastic, thank you for doing this update. FWIW, compressing VM images (some sparse files, some not) would take upwards of 45 minutes, which after this update, just takes a few minutes. root@releng2:/R2/vmimages # time xz -T 0 -k FreeBSD-11.0-CURRENT-amd64.qcow2 \ time xz -T 0 -k FreeBSD-11.0-CURRENT-amd64.raw; \ time xz -T 0 -k FreeBSD-11.0-CURRENT-amd64.vhd; \ time xz -T 0 -k FreeBSD-11.0-CURRENT-amd64.vmdk 1027.602u 40.376s 1:09.57 1535.1% 81+192k 0+19774io 0pf+0w 1032.978u 38.823s 1:08.17 1572.2% 81+192k 0+19696io 0pf+0w 1033.908u 38.593s 1:11.70 1495.8% 81+192k 0+19729io 0pf+0w 1091.749u 42.371s 1:04.27 1764.6% 81+192k 0+19751io 0pf+0w I meant to include that this is on a 48-core machine. Hm, I can't beleive you didn't use pxz ;) Anyway, having this in base, and not depending on external tool, is amazingly great. BTW, Rui, did you some comparative tests with pxz? -- Sincerely, D.Marck [DM5020, MCK-RIPE, DM3-RIPN] [ FreeBSD committer: ma...@freebsd.org ] *** Dmitry Morozovsky --- D.Marck --- Wild Woozle --- ma...@rinet.ru *** ___ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org
Re: svn commit: r278433 - in head: . contrib/xz contrib/xz/src/common contrib/xz/src/liblzma contrib/xz/src/liblzma/api contrib/xz/src/liblzma/api/lzma contrib/xz/src/liblzma/check contrib/xz/src/libl
On Mon, Feb 09, 2015 at 12:25:40PM +, Glen Barber wrote: On Mon, Feb 09, 2015 at 06:20:35AM +, Rui Paulo wrote: Author: rpaulo Date: Mon Feb 9 06:20:34 2015 New Revision: 278433 URL: https://svnweb.freebsd.org/changeset/base/278433 Log: Merge xz 5.2.0. This brings support for multi-threaded compression. This brings close N times faster compression where N is the number of CPU cores. Because of this, liblzma now depends on libthr. This is fantastic, thank you for doing this update. FWIW, compressing VM images (some sparse files, some not) would take upwards of 45 minutes, which after this update, just takes a few minutes. root@releng2:/R2/vmimages # time xz -T 0 -k FreeBSD-11.0-CURRENT-amd64.qcow2 \ time xz -T 0 -k FreeBSD-11.0-CURRENT-amd64.raw; \ time xz -T 0 -k FreeBSD-11.0-CURRENT-amd64.vhd; \ time xz -T 0 -k FreeBSD-11.0-CURRENT-amd64.vmdk 1027.602u 40.376s 1:09.57 1535.1% 81+192k 0+19774io 0pf+0w 1032.978u 38.823s 1:08.17 1572.2% 81+192k 0+19696io 0pf+0w 1033.908u 38.593s 1:11.70 1495.8% 81+192k 0+19729io 0pf+0w 1091.749u 42.371s 1:04.27 1764.6% 81+192k 0+19751io 0pf+0w I meant to include that this is on a 48-core machine. Glen pgpPQ0lQJDnAF.pgp Description: PGP signature
Re: svn commit: r278433 - in head: . contrib/xz contrib/xz/src/common contrib/xz/src/liblzma contrib/xz/src/liblzma/api contrib/xz/src/liblzma/api/lzma contrib/xz/src/liblzma/check contrib/xz/src/libl
On Feb 8, 2015, at 22:20, Rui Paulo rpa...@freebsd.org wrote: Author: rpaulo Date: Mon Feb 9 06:20:34 2015 New Revision: 278433 URL: https://svnweb.freebsd.org/changeset/base/278433 Log: Merge xz 5.2.0. This brings support for multi-threaded compression. This brings close N times faster compression where N is the number of CPU cores. Because of this, liblzma now depends on libthr. Gah, terrible wording. I bet I couldn't repeat the word this three times in a row if I wanted to... -- Rui Paulo ___ svn-src-all@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/svn-src-all To unsubscribe, send any mail to svn-src-all-unsubscr...@freebsd.org
