[bug #18396] stack size setrlimit call interacts badly with Solaris/x86 kernel bug
URL: http://savannah.gnu.org/bugs/?18396 Summary: stack size setrlimit call interacts badly with Solaris/x86 kernel bug Project: make Submitted by: smcpeak Submitted on: Tuesday 11/28/2006 at 20:16 Severity: 3 - Normal Item Group: Bug Status: None Privacy: Public Assigned to: None Open/Closed: Open Discussion Lock: Any Component Version: 3.81 Operating System: POSIX-Based Fixed Release: None ___ Details: I recently experienced the following mysterious phenomenon: * Running a program from the shell, it crashes (it dereferences NULL on purpose; that is all it does). * Invoking that same program from with a Makefile, it does not crash. Instead, it happily reads/writes page 0. After considerable investigation, I have discovered that the cause is two interacting issues: * There is a kernel bug in Solaris/x86 that causes page 0 to be mapped if the stack limit is 'unlimited'; see http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6374692 * GNU make was modified on 2001-09-06 (between 3.79.1 and 3.80) by Paul Eggert to Get rid of any avoidable limit on stack size (quote from ChangeLog). See the 'setrlimit' call in main.c. Consequently, whether a program is running under 'make' greatly affects how it behaves, as the child processes inherit the resource limits as well. While the kernel issue is clearly a bug, I think 'make' behavior is a misfeature as well. Generally one expects resource limits to not be silently changed by shells and shell-like programs such as 'make'. That 'make' does this is troubling; among other things, diagnosing the consequences is difficult (I investigated many other possible causes before finding it). The Solaris kernel bug is just one way such a silent change might be manifested. If 'make' needs to allocate a large amount (megabytes) of data, it would be better to do so on the heap, both from a portability standpoint (the stack size) and from a performance standpoint (it messes up the normally good locality of stack access). Alternatively, if it must allocate on the stack, then detecting and complaining about a too-low limit would be better in my opinion than silently changing it. It's easy to uncap the stack size explicitly in build scripts and whatnot when truly necessary. Output of uname -a: SunOS tainted.sf.coverity.com 5.10 Generic_118844-26 i86pc i386 i86pc Solaris Output of make -v: GNU Make 3.81 Copyright (C) 2006 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. This program built for i386-pc-solaris2.10 ___ Reply to this item at: http://savannah.gnu.org/bugs/?18396 ___ Message sent via/by Savannah http://savannah.gnu.org/ ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
[bug #18396] stack size setrlimit call interacts badly with Solaris/x86 kernel bug
Update of bug #18396 (project make): Item Group: Bug = Enhancement ___ Follow-up Comment #1: If 'make' needs to allocate a large amount (megabytes) of data, it would be better to do so on the heap, both from a portability standpoint (the stack size) and from a performance standpoint (it messes up the normally good locality of stack access). Alternatively, if it must allocate on the stack, then detecting and complaining about a too-low limit would be better in my opinion than silently changing it. It's easy to uncap the stack size explicitly in build scripts and whatnot when truly necessary. Unfortunately, none of the above is true. Make needs extra stack space because it makes extensive use of the alloca() function. It does not allocate huge chunks of memory on the stack: if large amounts of memory are needed they are allocated on the stack. alloca() is used for modest-sized memory chunks, to hold filenames and smaller strings (make is, at heart, a string manipulation program). Using heap, which requires a system call to get more memory, for all of these small allocations would be much less efficient than using the stack. Not to mention the overhead of having to allocate these memory segments to be used for a short time, then freeing them again, over and over. However, because make is also very recursive, even though no single alloca() call is very large it's quite possible for the entirety of the alloca() invocations for a complex makefile to use quite a bit of stack. Of course, there's no way to determine how much stack will be used a priori, since it depends entirely on the construction of the makefile, exactly which functions are invoked in which order, and even the current state of the build (whether various targets need to be rebuilt or not). Further, there is no portable way that I'm aware of to determine how much stack is left before the program runs out. Finally, there is no way to detect an out of stack error and exit gracefully with a warning as you suggest: the behavior of alloca() is undefined if you run out of stack space (it doesn't just return NULL as malloc() etc. do). So. In order to avoid the need for extra stack in GNU make all the invocations of alloca() would need to be rewritten to use xmalloc(), and those functions would need to be changed to be careful to free all that memory whenever the function exited to avoid memory leaks... with what must certainly be a decrease in performance (although of how much we can't be sure). On the other hand, as you point out, the problem on Solaris is a bug in the kernel which you can hardly blame GNU make for. However, your point about programs invoked by make inheriting the setrlimit() is definitely something that seems problematic. Perhaps GNU make could change the stack limit back to what it was after it forks but before it execs its child. I wonder what happens if you change a limit to something too small for the current processes resources? ___ Reply to this item at: http://savannah.gnu.org/bugs/?18396 ___ Message sent via/by Savannah http://savannah.gnu.org/ ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
[bug #18396] stack size setrlimit call interacts badly with Solaris/x86 kernel bug
Follow-up Comment #2, bug #18396 (project make): I wrote: if large amounts of memory are needed they are allocated on the stack Of course I meant on the _heap_ :-/. ___ Reply to this item at: http://savannah.gnu.org/bugs/?18396 ___ Message sent via/by Savannah http://savannah.gnu.org/ ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
RE: [bug #18396] stack size setrlimit call interacts badly with Solaris/x86 kernel bug
Using heap, which requires a system call to get more memory
(It doesn't affect the main point of Paul's reply but just for academic
interest) no it doesn't:
[EMAIL PROTECTED]:~/playpen$ cat ten-thousand-mallocs.c
#include stdlib.h
int main() {
for (int ii = 0; ii != 10 * 1000; ++ ii) {
free(malloc(1));
}
}
[EMAIL PROTECTED]:~/playpen$ g++ ten-thousand-mallocs.c
[EMAIL PROTECTED]:~/playpen$ strace ./a.out 21 | wc -l
152
[EMAIL PROTECTED]:~/playpen$
Even in less contrived applications, brk isn't called anything like as
often as malloc.
-
Martin's Outlook, BlueArc Engineering
___
Bug-make mailing list
Bug-make@gnu.org
http://lists.gnu.org/mailman/listinfo/bug-make
RE: [bug #18396] stack size setrlimit call interacts badly with Solaris/x86 kernel bug
On Tue, 2006-11-28 at 18:45 -0800, Martin Dorey wrote: Using heap, which requires a system call to get more memory (It doesn't affect the main point of Paul's reply but just for academic interest) no it doesn't: Even in less contrived applications, brk isn't called anything like as often as malloc. Certainly that's true... I didn't mean to suggest every call to malloc() resulted in a system call. But using the heap DOES require a system call to get memory... obviously the C runtime will grab memory in much larger chunks and maintains a free pool, with algorithms to handle fragmentation and coalescing of free blocks, etc. There are whole dissertations written on the most efficient ways to manage heap :). But (as I'm sure we both agree) alloca() is unquestionably more efficient (assuming you have an appropriate amount of stack). ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: plzzzzzzzzzz help
On Tue, 2006-11-28 at 03:48 -0800, rocky john wrote: I am amateur at linux .i am trying to install nagios-2.6.i went through documentations after executing commands make install -commandmode make install -config I got this problem :This program built for i386-redhat-linux-gnu.. My pc is i386 packages compatible..help me out As someone else pointed out, you should be asking for help in the nagios lists. May I suggest that when you ask them, you provide many more details including the exact command you entered, the precise hardware and operating system type and version you're using, the exact errors you got (cut and paste please) including a useful number of lines of context before the error. And most important: please use a subject line that is relevant and coherent! These days, spam is a huge percentage of all email and the open source mailing lists are no exception. In order to allow anyone to report problems and ask for help many OSS mailing lists don't require you to subscribe before posting... that means that someone has to go through and look through the spam to find the messages that should be forwarded to the list. Having a relevant subject that is immediately recognizable as not spam, vs. one that is quite spam-like, greatly decreases the chances of your email being unceremoniously dumped into the bit bucket and makes life a lot easier for us poor saps who have to sort it all out. Good luck! ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
Re: plzzzzzzzzzz help
These days, spam is a huge percentage of all email and the open source mailing lists are no exception. In order to allow anyone to report problems and ask for help many OSS mailing lists don't require you to subscribe before posting... that means that someone has to go through and look through the spam to find the messages that should be forwarded to the list. In the GNU Project, we advocate free software, not open source. Open source is the slogan of a movement that was formed specifically to reject (and hush up) our philosophy, so we never use that term here except to explain the difference between the two movements. See http://www.gnu.org/philosophy/free-software-for-freedom.html. As representatives of the GNU project it makes it even more important that we clearly state that we and the work we do is not part of a movement that was created to silence us. Cheers. ___ Bug-make mailing list Bug-make@gnu.org http://lists.gnu.org/mailman/listinfo/bug-make
