Re: Setting RLIMIT_STACK
On Sat, 2010-07-17 at 02:22 -0300, Alberto Bertogli wrote: > So I was wondering what was the reason for that rlimit, and if perhaps > it could be avoided to prevent this behaviour change. It was added a number of years ago by Paul Eggert, because make makes extensive use of alloca() and some larger, more complex makefiles might run into problems. Probably there should be an effort to switch to heap for anything that might get large and reserve alloca() usage just for things we know for a fact will not get too large, but that hasn't been done. -- --- Paul D. Smith Find some GNU make tips at: http://www.gnu.org http://make.mad-scientist.net "Please remain calm...I may be mad, but I am a professional." --Mad Scientist ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: Setting RLIMIT_STACK
On Sat, 2010-07-17 at 02:22 -0300, Alberto Bertogli wrote: > Chasing a bug, I noticed that make sets RLIMIT_STACK to RLIM_INFINITY. > > In Linux (since 2.6.25), that actually triggers a very subtle behaviour > change in the way the kernel manages the memory layout. While most > applications are oblivious to it, some are not and can cause issues that > are not so easy to see. > > The layouts are described at http://lwn.net/Articles/90311/, and while > Linux usually uses the new one (described there as "top-down mmap() > method"), when RLIMIT_STACK is set to RLIM_INFINITY it falls back to the > old one (the one described first). Hi Alberto. I was looking into this more. Although GNU make does set RLIMIT_STACK to infinity while it, itself, is running, it resets the stack size back to the original value it had when it was started before it runs any commands. That is, in job.c after we fork() to start the command we call setrlimit() again to reset the stack back to what it was, before we exec() the command we're about to run. So, I'm not sure I fully understand the implications here. Are you saying that this resetting of the stack limit back to what it was does not reset the memory layout back to using the new method in make's children, and this causes different behavior in make's children? Or that the change somehow subtly changes the behavior of _make_ in a less-than-optimal way? I'd like to understand this so we can decide how to proceed for make 3.82 which will be out very, very shortly (so if we're going to remove this we need to make that decision right now, basically). -- --- Paul D. Smith Find some GNU make tips at: http://www.gnu.org http://make.mad-scientist.net "Please remain calm...I may be mad, but I am a professional." --Mad Scientist ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: Setting RLIMIT_STACK
On Sun, 2010-07-18 at 14:40 -0300, Alberto Bertogli wrote: > The layout is decided by the kernel early, when the new process is being > created in execve(), and cannot be changed while running. I did suspect as much: it seemed impossible (or at least highly unlikely) for this to be changed during runtime. OK, seems to be OK for the next release. Thanks! -- --- Paul D. Smith Find some GNU make tips at: http://www.gnu.org http://make.mad-scientist.net "Please remain calm...I may be mad, but I am a professional." --Mad Scientist ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: Setting RLIMIT_STACK
On Sun, Jul 18, 2010 at 12:54:56PM -0400, Paul Smith wrote: > On Sat, 2010-07-17 at 02:22 -0300, Alberto Bertogli wrote: > > Chasing a bug, I noticed that make sets RLIMIT_STACK to RLIM_INFINITY. > > > > In Linux (since 2.6.25), that actually triggers a very subtle behaviour > > change in the way the kernel manages the memory layout. While most > > applications are oblivious to it, some are not and can cause issues that > > are not so easy to see. > > > > The layouts are described at http://lwn.net/Articles/90311/, and while > > Linux usually uses the new one (described there as "top-down mmap() > > method"), when RLIMIT_STACK is set to RLIM_INFINITY it falls back to the > > old one (the one described first). > > Hi Alberto. I was looking into this more. Although GNU make does set > RLIMIT_STACK to infinity while it, itself, is running, it resets the > stack size back to the original value it had when it was started before > it runs any commands. Thanks for looking into it! > That is, in job.c after we fork() to start the command we call > setrlimit() again to reset the stack back to what it was, before we > exec() the command we're about to run. > > So, I'm not sure I fully understand the implications here. > > Are you saying that this resetting of the stack limit back to what it > was does not reset the memory layout back to using the new method in > make's children, and this causes different behavior in make's children? > Or that the change somehow subtly changes the behavior of _make_ in a > less-than-optimal way? Layout changes affect make's children, but not make itself. The layout is decided by the kernel early, when the new process is being created in execve(), and cannot be changed while running. So changing it after fork() but prior to execve(), as you said, should be ok. However, I'm not seeing this behaviour in make 3.81 (Ubuntu's, not my own). Peeking around the source browser I notice that the change that introduced the setrlimit() you describe is commit 1.191, dated 2008-11-30, which is post-3.81. Seeing that, I tried the latest alpha release (3.81.90), and it works as expected! So I can confirm that the issue I was seeing has already been fixed in the new release, and as far as I can see there are no further changes needed to address it. Thanks a lot, Alberto ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: Setting RLIMIT_STACK
On Sun, Jul 18, 2010 at 02:40:19PM -0300, Alberto Bertogli wrote: > Peeking around the source browser I notice that the change that introduced > the setrlimit() you describe is commit 1.191, dated 2008-11-30, which is > post-3.81. > > Seeing that, I tried the latest alpha release (3.81.90), and it works as > expected! I've just read the thread about this in the general list, sorry I missed it before (if anyone is wondering, I'm referring to "Weird performance problem", http://thread.gmane.org/gmane.comp.gnu.make.general/8974). While I talk to Leandro often, I didn't know there was a discussion about this bug going on over there when I posted these mails (my fault). I'm sorry if this caused you any trouble, and I'm glad it's already fixed. Thanks again, Alberto ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: Setting RLIMIT_STACK
On Mon, 2010-07-19 at 11:29 +0200, Edward Welbourne wrote: > > Probably there should be an effort to switch to heap for anything that > > might get large and reserve alloca() usage just for things we know for a > > fact will not get too large, but that hasn't been done. > > ... and anywhere you use a scanf variant, glibc is also using > alloca(), without knowing any bound: > http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=553206 Yes, unpleasant for sure. Luckly GNU make doesn't use *scanf() much at all and were it does only in fairly well-defined situations. -- --- Paul D. Smith Find some GNU make tips at: http://www.gnu.org http://make.mad-scientist.net "Please remain calm...I may be mad, but I am a professional." --Mad Scientist ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: Setting RLIMIT_STACK
> Probably there should be an effort to switch to heap for anything that > might get large and reserve alloca() usage just for things we know for a > fact will not get too large, but that hasn't been done. ... and anywhere you use a scanf variant, glibc is also using alloca(), without knowing any bound: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=553206 Eddy. ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
