Re: Bug#510624: ITP: pigz -- Parallel Implementation of GZip
Josselin Mouette wrote: Le lundi 05 janvier 2009 à 00:58 +0100, Samuel Thibault a écrit : You mean Scheduler Activations? There's a patch against linux 2.4 ;) We're definitely diving into OS research :) Well it would be nice if things that was research at the time of Linux 2.4 could have turned into usable code now :) Hum... Scheduler Activations for 2.4 did not handle the case described here. User application were notified when one of their (user) thread did a blocking system call (so that the application can run another user thread on the available core/kernel thread). But notifying the application for each kernel preemption decision was too costly and not implemented on Linux. Similar behavior have been proposed for 2.6 kernel (I remember LWN articles about asynchronous system call such as http://lwn.net/Articles/223899/) I'm confident that Scheduler Activations can be build upon such a proposition. About integration in upstream, it is very difficult to tell upstream that user threads are interesting (even if it is the case in HPC). And the current design of ntpl on x86 is a shame: allowing a direct access to TLS through the segment register makes very difficult to write a user-(or mixed-)level thread library for Linux, keeping the same ABI (and libc). See #328183 for more info on that. So, maintaining such patches (kernel, glibc, ntpl, ...) would need really lot of time. Vincent -- Vincent Danjean GPG key ID 0x9D025E87 vdanj...@debian.org GPG key fingerprint: FC95 08A6 854D DB48 4B9A 8A94 0BF7 7867 9D02 5E87 Unofficial pacakges: http://www-id.imag.fr/~danjean/deb.html#package APT repo: deb http://perso.debian.org/~vdanjean/debian unstable main -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: Bug#510624: ITP: pigz -- Parallel Implementation of GZip
Le lundi 05 janvier 2009 à 00:58 +0100, Samuel Thibault a écrit : You mean Scheduler Activations? There's a patch against linux 2.4 ;) We're definitely diving into OS research :) Well it would be nice if things that was research at the time of Linux 2.4 could have turned into usable code now :) More seriously, I wouldn't see the kernel being able to start threads, that's painful to maintain (that's why the LinuxActivations patch hasn't been maintained). I'd rather see a way for the process to dynamically know how many processors it can currently expect (something like a /proc, /sys, /dev or whatever fd with notification), and arrange things according to that (starting/stopping worker threads for instance). The idea of making the kernel itself start new threads is probably not so great, but a simple mechanism to let a process know when the number of available cores for it has changed would do the trick without adding too much complexity. -- .''`. : :' : We are debian.org. Lower your prices, surrender your code. `. `' We will add your hardware and software distinctiveness to `-our own. Resistance is futile. signature.asc Description: Ceci est une partie de message numériquement signée
Re: Bug#510624: ITP: pigz -- Parallel Implementation of GZip
On Sat, Jan 03, 2009 at 09:57:33PM +0100, Eduard Bloch wrote: PS: I plan to hack it a little bit and use syssconf function on Debian systems to determine the real number of CPU cores (#x) since pigz's default value is 8 which is much more than home systems have nowadays, and the performance isn't getting (much) better with a constant number of idle threads, they just consume more memory. Although sysconf() can tell you the total number of cores and the number of cores that are online in your system, it does not tell you how many cores are available for your program. It's better to use sched_getaffinity() to get the set of CPU (cores) available to your program. And if you know better than the OS, use sched_setaffinity() to bind each thread to it's own core. -- Met vriendelijke groet / with kind regards, Guus Sliepen g...@debian.org signature.asc Description: Digital signature
Re: Bug#510624: ITP: pigz -- Parallel Implementation of GZip
#include hallo.h * Guus Sliepen [Sun, Jan 04 2009, 10:45:23AM]: On Sat, Jan 03, 2009 at 09:57:33PM +0100, Eduard Bloch wrote: PS: I plan to hack it a little bit and use syssconf function on Debian systems to determine the real number of CPU cores (#x) since pigz's default value is 8 which is much more than home systems have nowadays, and the performance isn't getting (much) better with a constant number of idle threads, they just consume more memory. Although sysconf() can tell you the total number of cores and the number of cores that are online in your system, it does not tell you how many cores are available for your program. It's better to use sched_getaffinity() to get the set of CPU (cores) available to your program. And if you know better than the OS, use sched_setaffinity() to bind each thread to it's own core. Sounds like a plan, but I don't feel very comfortable to do that in the Debian package. Let me explain why: - sched_setaffinity method seems to be Linux specific - though it might be beneficial to pinpoint each thread to the same CPU, I don't think (*assumption*) that the data resides long enough in the CPU cache anyway so we wouldn't win much - I have not used it before but I used the sysconf method in portable apps (i.e. in cloop-utils), sysconf method is also used in pbzip2 - it's hard to imagine environments with big difference between count(cores) and count(available cores) - the code change is minimal, see below @Marc: please check the -s option, it's called -S in gzip (upper case). What's the reason for the different case? Regards, Eduard. --- pigz-2.1.4.orig/pigz.c +++ pigz-2.1.4/pigz.c @@ -2795,8 +2795,13 @@ #ifdef NOTHREAD procs = 1; #else + +#ifdef _SC_NPROCESSORS_ONLN +procs=sysconf(_SC_NPROCESSORS_ONLN); +#else procs = 8; -#endif +#endif /* _SC_NPROCESSORS_ONLN */ +#endif /* NOTHREAD */ size = 131072UL; rsync = 0; /* don't do rsync blocking */ dict = 1; /* initialize dictionary each thread */ -- Naja, Garbage Collector eben. Holt den Müll sogar vom Himmel. (Heise Trollforum über Java in der Flugzeugsteuerung) signature.asc Description: Digital signature
Re: Bug#510624: ITP: pigz -- Parallel Implementation of GZip
Le dimanche 04 janvier 2009 à 15:49 +0100, Eduard Bloch a écrit : Sounds like a plan, but I don't feel very comfortable to do that in the Debian package. Let me explain why: - sched_setaffinity method seems to be Linux specific How is that a problem? You only need to use it in Linux builds. - it's hard to imagine environments with big difference between count(cores) and count(available cores) It’s already the case in HPC environments, and CPU pinning is certainly going to be used more widely as the number of cores increases. -- .''`. : :' : We are debian.org. Lower your prices, surrender your code. `. `' We will add your hardware and software distinctiveness to `-our own. Resistance is futile. signature.asc Description: Ceci est une partie de message numériquement signée
Re: Bug#510624: ITP: pigz -- Parallel Implementation of GZip
Josselin Mouette, le Sun 04 Jan 2009 16:07:25 +0100, a écrit : Le dimanche 04 janvier 2009 à 15:49 +0100, Eduard Bloch a écrit : Sounds like a plan, but I don't feel very comfortable to do that in the Debian package. Let me explain why: - sched_setaffinity method seems to be Linux specific How is that a problem? You only need to use it in Linux builds. And revert to using sysconf() in the cases when you're not on Linux. - it's hard to imagine environments with big difference between count(cores) and count(available cores) It’s already the case in HPC environments, and CPU pinning is certainly going to be used more widely as the number of cores increases. And that's a shame. Linux shouldn't be so happy to move tasks between CPUs... Samuel -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: Bug#510624: ITP: pigz -- Parallel Implementation of GZip
Le dimanche 04 janvier 2009 à 23:45 +0100, Samuel Thibault a écrit : It’s already the case in HPC environments, and CPU pinning is certainly going to be used more widely as the number of cores increases. And that's a shame. Linux shouldn't be so happy to move tasks between CPUs... Actually it doesn’t. Since CPU affinity was included (IIRC in 2.6.16) it is much less prone to move tasks, and the performance impact of not using CPU pinning is small. Still, it is better to use CPU pinning since you often want finer control than that, and that’s especially true in multi-user environments where resources can be sub-host. -- .''`. : :' : We are debian.org. Lower your prices, surrender your code. `. `' We will add your hardware and software distinctiveness to `-our own. Resistance is futile. signature.asc Description: Ceci est une partie de message numériquement signée
Re: Bug#510624: ITP: pigz -- Parallel Implementation of GZip
Josselin Mouette, le Mon 05 Jan 2009 00:20:42 +0100, a écrit : Samuel Thibault, le Sun 04 Jan 2009 23:45:22 +0100, a écrit : It’s already the case in HPC environments, and CPU pinning is certainly going to be used more widely as the number of cores increases. And that's a shame. Linux shouldn't be so happy to move tasks between CPUs... Actually it doesn’t. Since CPU affinity was included (IIRC in 2.6.16) it is much less prone to move tasks, and the performance impact of not using CPU pinning is small. Things have improved, yes. Still, it is better to use CPU pinning since you often want finer control than that, and that’s especially true in multi-user environments where resources can be sub-host. Sure, but that should be only a user-explicitely-wanting thing. I would really not like to see pigz systematically bind threads. What if I e.g. want to run several pigz processes at the same time because I have a lot of cores and a lot of files (I guess pigz doesn't scale so much)? Ideally, we should just let Linux manage everything, i.e. put related threads together on the same dies, balancing the load according to the observed behavior, which can vary a lot depending on the latency of reading files, the time to compress, etc. Samuel -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: Bug#510624: ITP: pigz -- Parallel Implementation of GZip
On 01/04/09 17:20, Josselin Mouette wrote: Le dimanche 04 janvier 2009 à 23:45 +0100, Samuel Thibault a écrit : It’s already the case in HPC environments, and CPU pinning is certainly going to be used more widely as the number of cores increases. And that's a shame. Linux shouldn't be so happy to move tasks between CPUs... Actually it doesn’t. Since CPU affinity was included (IIRC in 2.6.16) it is much less prone to move tasks, and the performance impact of not using CPU pinning is small. Still, it is better to use CPU pinning since you often want finer control than that, and that’s especially true in multi-user environments where resources can be sub-host. Wouldn't it be better to bind a process to a chip, rather than a core, so that you don't run into cases where, after many processes terminate, you are left with most of the busy processes pinned to a single busy core while the others are mostly unused? -- Ron Johnson, Jr. Jefferson LA USA I like my women like I like my coffee - purchased at above-market rates from eco-friendly organic farming cooperatives in Latin America. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: Bug#510624: ITP: pigz -- Parallel Implementation of GZip
Le lundi 05 janvier 2009 à 00:38 +0100, Samuel Thibault a écrit : Sure, but that should be only a user-explicitely-wanting thing. I would really not like to see pigz systematically bind threads. What if I e.g. want to run several pigz processes at the same time because I have a lot of cores and a lot of files (I guess pigz doesn't scale so much)? Ideally, we should just let Linux manage everything, i.e. put related threads together on the same dies, balancing the load according to the observed behavior, which can vary a lot depending on the latency of reading files, the time to compress, etc. There is probably a missing piece here. If you start several pigz processes, the kernel only sees processes starting a lot of threads, and processes only see a given number of cores. There is no interface that allows a process to specify how to start more threads, giving the kernel the opportunity to start them as it sees fit given the available number of cores. -- .''`. : :' : We are debian.org. Lower your prices, surrender your code. `. `' We will add your hardware and software distinctiveness to `-our own. Resistance is futile. signature.asc Description: Ceci est une partie de message numériquement signée
Re: Bug#510624: ITP: pigz -- Parallel Implementation of GZip
Ron Johnson, le Sun 04 Jan 2009 17:40:08 -0600, a écrit : On 01/04/09 17:20, Josselin Mouette wrote: Still, it is better to use CPU pinning since you often want finer control than that, and that’s especially true in multi-user environments where resources can be sub-host. Wouldn't it be better to bind a process to a chip, rather than a core, so that you don't run into cases where, after many processes terminate, you are left with most of the busy processes pinned to a single busy core while the others are mostly unused? That's precisely the kind of thing that makes it better to just leave it up to Linux. The case of HPC is quite particular in that you usually really precisely control your computation. In the case of general-purpose tools, I would really not recommend it. Samuel -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: Bug#510624: ITP: pigz -- Parallel Implementation of GZip
Josselin Mouette, le Mon 05 Jan 2009 00:47:02 +0100, a écrit : There is probably a missing piece here. If you start several pigz processes, the kernel only sees processes starting a lot of threads, and processes only see a given number of cores. There is no interface that allows a process to specify how to start more threads, giving the kernel the opportunity to start them as it sees fit given the available number of cores. You mean Scheduler Activations? There's a patch against linux 2.4 ;) We're definitely diving into OS research :) More seriously, I wouldn't see the kernel being able to start threads, that's painful to maintain (that's why the LinuxActivations patch hasn't been maintained). I'd rather see a way for the process to dynamically know how many processors it can currently expect (something like a /proc, /sys, /dev or whatever fd with notification), and arrange things according to that (starting/stopping worker threads for instance). Samuel -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: Bug#510624: ITP: pigz -- Parallel Implementation of GZip
On Mon, Jan 5, 2009 at 8:49 AM, Samuel Thibault samuel.thiba...@ens-lyon.org wrote: That's precisely the kind of thing that makes it better to just leave it up to Linux. The case of HPC is quite particular in that you usually really precisely control your computation. In the case of general-purpose tools, I would really not recommend it. I imagine leaving it up to Linux would mean it would work better on clustered systems like Kerrighed: http://www.kerrighed.org/ (hi KiBi) -- bye, pabs http://wiki.debian.org/PaulWise -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#510624: ITP: pigz -- Parallel Implementation of GZip
Package: wnpp Severity: wishlist Owner: Eduard Bloch bl...@debian.org * Package name: pigz Version : 2.1.4 Upstream Author : Mark Adler mad...@alumni.caltech.edu * URL : http://www.example.org/ * License : ZLib license Programming Lang: C Description : Parallel Implementation of GZip pigz, which stands for Parallel Implementation of GZip, is a fully functional replacement for gzip that exploits multiple processors and multiple cores to the hilt when compressing data. PS: I plan to hack it a little bit and use syssconf function on Debian systems to determine the real number of CPU cores (#x) since pigz's default value is 8 which is much more than home systems have nowadays, and the performance isn't getting (much) better with a constant number of idle threads, they just consume more memory. Regards, Eduard. -- System Information: Debian Release: 5.0 APT prefers unstable APT policy: (500, 'unstable'), (1, 'experimental') Architecture: amd64 (x86_64) -- Naja, Garbage Collector eben. Holt den Müll sogar vom Himmel. (Heise Trollforum über Java in der Flugzeugsteuerung) -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
