Re: Setting memory allocators for library functions.
Matt Dillon wrote: Allowing a program to run the OS itself out of VM, with or without overcommit, is (being generous) just plain dumb. I'm not a fan of either (overcommit or non-), I can see advantages with both (seeing that Solaris, which I happen to work with, has one and FreeBSD the other), but your last remark does beg an answer: In a non-dedicated environment (ie a general-purpose Unix machine), it's the mix of applications that brings down your memory, not a single one. In such a situation I can imagine synchronous information to the effect "you're out of swapable memory" to be practical (that's the way Solaris implements it). I haven't thought this out in detail, but I also imagine it easier to handle ENOMEM than SIGDANGER, because of the synchronous nature of the first versus the asynchronous nature of the second. just my 2 euro cents Michael -- Michael Schuster / [EMAIL PROTECTED] Sun Microsystems GmbH / (+49 89) 46008-2974 | x62974 Sonnenallee 1, D-85551 Kirchheim-Heimstetten Recursion, n.: see 'Recursion' To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
Tony Finch [EMAIL PROTECTED] writes: What about setrlimit(RLIMIMT_DATA)? Yep, I'd forgotten about that. Malloc() will return NULL if you hit your data size limit. DES -- Dag-Erling Smorgrav - [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
On Tue, Feb 27, 2001 at 10:39:13PM -0800, Julian Elischer wrote: no, something specifically designed around kernel type of actions. declarations of "physical pointer", "kvm pointer" "User Pointer" for example, and being able to declare a structure (not 'struct') and say "this list is 'per process'" and have the list head automatically in the proc struct without haviong to add it there.. i.e backwards from today.. Rumor has it that MS has several compiler extensions, just for supporting their kernel. Some of what you say above could be built on top of the compiler, declaratively. Language support works well in cases where writing the same code by hand is tedious and error prone or down right ugly - like several hundred if (foo = null) return blah checks. -Arun To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
For your collective amusement, here's a post that talks about how OS/2 handles memory allocation. DosAllocMem() has a flags argument, and one of the flags requests the OS to actually commit the memory. http://w3.hethmon.com/os2isp/1998/Apr/Msgs/l2w96957.html http://www.stidolph.com/os2api/Dos/DosAllocMem.html So even IBM must have thought it not to be such a bad idea. -- Jos Backus _/ _/_/_/"Modularity is not a hack." _/ _/ _/-- D. J. Bernstein _/ _/_/_/ _/ _/ _/_/ [EMAIL PROTECTED] _/_/ _/_/_/use Std::Disclaimer; To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
: things work. Then try coding conditionals all the way through to fix : it... and don't forget you need to propogate the error condition back : up the procedure chain too so the original caller knows why it failed. : :So, it all comes down to reimplementing the UNIX kernel in a language :that supports exceptions, just like Linus suggested :) : : -Arun Not really. UNIX works just fine, it gives you plenty sufficient control over your environment and you can write your programs pretty much in whatever language you like. But no amount of OS control will magically save a badly written program from itself. The best you can hope for is to reduce the collateral damage by setting appropriate resource limits. Allowing a program to run the OS itself out of VM, with or without overcommit, is (being generous) just plain dumb. -Matt To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
Matt Dillon wrote: Said application was poorly written, then. Even on solaris if you The only reason the application was "poorly written" is the overcommit architecture. actually run the system out of memory you can blow up other unrelated processes. To depend on that sort of operation is just plain dumb. Not at all. You can fill all memory on Solaris and it will work just fine. Go ahead and try it, if you doubt me. :I'll give you one more example. Protocol validation. It is often :impossible to test all possible permutations of a protocol's dialog, but :being able to go as deep as possible on execution tree and then, when :you go out of memory, giving up on that path, backing down and :continuing elsewhere let you get a partial validation, which is not :enough to prove a protocol is correct but might well be enough to prove :it is incorrect. This is a real application, and one in which an out of :memory condition is not only handled but even expected. This has nothing to do with memory overcommit. Nothing at all. What is your definition of out-of-memory? When swap runs out, or when the system starts to thrash? What is the point of running a scientific When a memory request cannot be satisfied. Swap runs out, it would seem. calculation if the machine turns into a sludge pile and would otherwise cause the calculation to take years to complete instead of days? It doesn't trash. The memory is filled with backtracking information. Memory in active use at any time is rather small. You've got a whole lot more issues to deal with then simple memory overcommit, and you are ignoring them completely. Not at all. I'm giving you an example of an application which depends on non overcommitting and _works_ on such architectures. :And, of course, those whose infrastructure depends on a malloc() :returning NULL indicating the heap is full will not work on FreeBSD. :(sarcasmYou do recall that many of these languages are written in C, :don't you?/sarcasm) Bullshit. If you care, a simple wrapper will do what you want. Modern systems tend to have huge amounts of swap. Depending on malloc to Huge amounts of swaps is not a given. You are assuming a hardware setup to fit your theory. fail with unbounded resources in an overcommit OR a non-overcommit case is stupid, because the system will be thrashing heavily long before it even gets to that point. Allocating memory does not trash the system. A rather large number of pages in active use does, and this is not necessarily the case at all. Depending on malloc() to fail by setting an appropriate datasize limit resource is more reasonable, and malloc() does work as expected if you do that. I completely agree that setting datasize limit is more reasonable, but that does not prevent an application from being killed if the system does run out of memory. I think that if the system runs out of memory, you don't have enough memory. That datasize limits must be used to ensure desired behavior. But this is a _preference_. On Solaris, depending on non overcommitting of memory is possible, and some do prefer it that way. :It has everything to do with overcommit. In this particular case, not :only there _is_ something to do when the out of memory condition arise, :but the very algorithm depends on it arising. It has nothing to do with overcommit. You are confusing overcommit with hard-datasize limits, which can be set with a simple 'limit' command. Unless I want it to grab all available memory. : :Garbage Collection: Algorithms for Automatic Memory Management, Richard :Jones and Rafael Lins. Bullshit is what you just said. None of which requires overcommit. None of which would actually work in a real-world situation with or without overcommit if you do not hard-limit the memory resource for the program in the first place. If you ever bother to check the reference, you'll see that many of these algorithms were implemented and used in real world systems. You are again making the mistake of assuming that not having overcommit will magically solve all your problems. It doesn't even come close. No, *YOU* keep insisting that we assume that. The assumption is rather different: *with* overcommit the problems *cannot* be solved (except by using datasize limit, which I think it's entirely reasonable, but some don't). You think these garbage collection algorithms work by running the system out of VM and then backing off? That's pure nonsense. I don't "think" anything. I'm reporting facts. Many algorithms do work that way, whether you think they are non-sense or not. -- Daniel C. Sobral(8-DCS) [EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED] Acabou o hipismo-arte. Mas a desculpa brasileira mais ouvida em Sydney e' que nao tem mais cavalo bobo por ai'. To
Re: Setting memory allocators for library functions.
Arun Sharma wrote: On 26 Feb 2001 18:56:18 +0100, Matt Dillon [EMAIL PROTECTED] wrote: Ha. Right. Go through any piece of significant code and just see how much goes flying out the window because the code wants to simply assume things work. Then try coding conditionals all the way through to fix it... and don't forget you need to propogate the error condition back up the procedure chain too so the original caller knows why it failed. So, it all comes down to reimplementing the UNIX kernel in a language that supports exceptions, just like Linus suggested :) I've often considered writing a language SPECIFICALLY for writing the kernel. (no other uses) I mean it basically uses the same mechaninsims over and over and over again... linked lists, hash tables, nested loops, etc.etc. I'd like a language that lets me define the module I'm writing, define the way it should behave, and let the boring code be taken care of by itelf :-) -Arun To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message -- __--_|\ Julian Elischer / \ [EMAIL PROTECTED] ( OZ) World tour 2000-2001 --- X_.---._/ v To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
Daniel, you don't have to believe me, I really don't care. I hear this argument once or twice a year and those of who actually have experience (except for Terry, who always likes to argue theory) already know what the reality is. Come back in a year or two after you've wasted a thousand man hours trying to implement your little beast, tried to use it for something non trivial, and found that it doesn't work the way you expect. I've heard the solaris argument before two, though until you came along I didn't actually hear someone try to propound that running a machine out of memory on purpose wouldn't have any unwanted side effects. That's a new one.. and a good laugh. But it isn't reality. -Matt To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
Dag-Erling Smorgrav [EMAIL PROTECTED] wrote: This is all academic since FreeBSD does memory overcommit, so unless you run out of address space for your process before you run out of actual memory and/or swap (not likely, but quite possible) malloc() will never return NULL and you won't know a thing until you dirty one page too many and segfault. What about setrlimit(RLIMIMT_DATA)? Tony. -- f.a.n.finch[EMAIL PROTECTED][EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: [hackers] Re: Setting memory allocators for library functions.
Peter Seebach [EMAIL PROTECTED] wrote: David Gilbert writes: Due to the bloat of the OS and Motif and other such things, they required simply amazing amounts of swap just to run. Well, to some extent, I have to wonder why all these pages are being requested if they aren't being used... fork() with big data segments that cause swap to be reserved in case of a copy-on-write. The 2GB of swap is never actually used, but you still have to have it. Tony. -- f.a.n.finch[EMAIL PROTECTED][EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: [hackers] Re: Setting memory allocators for library functions.
"Tony" == Tony Finch [EMAIL PROTECTED] writes: Well, to some extent, I have to wonder why all these pages are being requested if they aren't being used... Tony fork() with big data segments that cause swap to be reserved Tony in case of a copy-on-write. The 2GB of swap is never Tony actually used, but you still have to have it. If the information in the data segment is going to be updated then you have to have writable backing store. If, however, that data is never going to be changed, it should be declared in the program as read-only data. The kernel VM system should not be working around applications that don't declare their data arena correctly. --lyndon To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
"Daniel C. Sobral" [EMAIL PROTECTED] wrote: Matt Dillon wrote: What is the point of running a scientific calculation if the machine turns into a sludge pile and would otherwise cause the calculation to take years to complete instead of days? It doesn't trash. The memory is filled with backtracking information. Memory in active use at any time is rather small. I've read articles about programs which use GC which have a small working set, although it is constantly changing (I've heard of programs allocating megabytes a second). The OS would have to swap out the stale pages if the program's total memory use exceeds RAM, and when the GC finally runs it will take forever and thrash swap like there's no tomorrow. Tony. -- f.a.n.finch[EMAIL PROTECTED][EMAIL PROTECTED] THAMES: EAST OR SOUTHEAST 3 OR 4. RAIN. MODERATE OR GOOD. To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
Tony Finch wrote: It doesn't trash. The memory is filled with backtracking information. Memory in active use at any time is rather small. I've read articles about programs which use GC which have a small working set, although it is constantly changing (I've heard of programs allocating megabytes a second). The OS would have to swap out the stale pages if the program's total memory use exceeds RAM, and when the GC finally runs it will take forever and thrash swap like there's no tomorrow. It depends a lot on how allocation is made and the characteristics of the reachable objects graph. If the likeness of a reachable object existing in each page isn't small, it will trash indeed. On the other hand, if most objects created have very short lifetimes and a generational technique is used to move objects in the working set, then few pages of the swap will need to be swapped in for tracing, and the rest can be outright discarded without ever swaping them in. On the gripping hand, FreeBSD on configurations without swap is not unheard of by any means either, and overcommitting is still a problem in these configurations. -- Daniel C. Sobral(8-DCS) [EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED] I think you are delusional, but that is OK. Its part of your natural charm! To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: [hackers] Re: Setting memory allocators for library functions.
Lyndon Nerenberg [EMAIL PROTECTED] wrote: If the information in the data segment is going to be updated then you have to have writable backing store. If, however, that data is never going to be changed, it should be declared in the program as read-only data. The kernel VM system should not be working around applications that don't declare their data arena correctly. In most cases it is impossible to declare the data read-only because it originally had to be read-write and you can't change its attributes later. This is always the case for malloc(). Many applications do a load of initialization that requires read-write memory then never write to that memory again; when they fork the OS still has to account for that memory twice even if it is going to be immediately discarded by an exec(). An even more exaggerated example is Apache which forks a load of times then hangs around for ages. Tony. -- f.a.n.finch[EMAIL PROTECTED][EMAIL PROTECTED] MALIN: NORTHEAST 7 TO SEVERE GALE 9, OCCASIONALLY STORM 10 IN SOUTHEAST AT FIRST, DECREASING 5. SNOW SHOWERS. GOOD. To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: [hackers] Re: Setting memory allocators for library functions.
In message [EMAIL PROTECTED], [EMAIL PROTECTED] writes: In most cases it is impossible to declare the data read-only because it originally had to be read-write and you can't change its attributes later. mprotect(2). To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: [hackers] Re: Setting memory allocators for library functions.
Drew Eckhardt [EMAIL PROTECTED] wrote: In message [EMAIL PROTECTED], [EMAIL PROTECTED] writes: In most cases it is impossible to declare the data read-only because it originally had to be read-write and you can't change its attributes later. mprotect(2). If it's available at all, mprotect() is often limited to memory obtained with mmap(), i.e. not malloc(). Not great for portability. Tony. -- f.a.n.finch[EMAIL PROTECTED][EMAIL PROTECTED] FORTH: NORTHEAST 6 TO GALE 8 DECREASING 5 OR 6. SLEET. MAINLY MODERATE. To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: [hackers] Re: Setting memory allocators for library functions.
Tony Finch wrote: Drew Eckhardt [EMAIL PROTECTED] wrote: In message [EMAIL PROTECTED], [EMAIL PROTECTED] writes: In most cases it is impossible to declare the data read-only because it originally had to be read-write and you can't change its attributes later. mprotect(2). If it's available at all, mprotect() is often limited to memory obtained with mmap(), i.e. not malloc(). Not great for portability. FreeBSD malloc() calls mmap() as AFAIK many (if not all) malloc() implementations. Maxime -- Don't be fooled by cheap finnish imitations ; BSD is the One True Code Key fingerprint = F9B6 1D5A 4963 331C 88FC CA6A AB50 1EF2 8CBE 99D6 Public Key : http://www.epita.fr/~henrio_m/ To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: [hackers] Re: Setting memory allocators for library functions.
Maxime Henrion [EMAIL PROTECTED] wrote: Tony Finch wrote: If it's available at all, mprotect() is often limited to memory obtained with mmap(), i.e. not malloc(). Not great for portability. FreeBSD malloc() calls mmap() as AFAIK many (if not all) malloc() implementations. FreeBSD malloc() uses sbrk() for memory that it returns to the application and mmap() for book-keeping memory (the page index). Tony. -- f.a.n.finch[EMAIL PROTECTED][EMAIL PROTECTED] GERMAN BIGHT: SOUTHEAST 4 OR 5 BACKING NORTHEAST 5 OR 6. SLEET. MODERATE OR POOR. To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
Julian Elischer wrote: Arun Sharma wrote: On 26 Feb 2001 18:56:18 +0100, Matt Dillon [EMAIL PROTECTED] wrote: Ha. Right. Go through any piece of significant code and just see how much goes flying out the window because the code wants to simply assume things work. Then try coding conditionals all the way through to fix it... and don't forget you need to propogate the error condition back up the procedure chain too so the original caller knows why it failed. So, it all comes down to reimplementing the UNIX kernel in a language that supports exceptions, just like Linus suggested :) I've often considered writing a language SPECIFICALLY for writing the kernel. (no other uses) I mean it basically uses the same mechaninsims over and over and over again... linked lists, hash tables, nested loops, etc.etc. I'd like a language that lets me define the module I'm writing, define the way it should behave, and let the boring code be taken care of by itelf :-) Oh, like C++ STL? /me ducks and runs, trying not to gag. -- "Where am I, and what am I doing in this handbasket?" Wes Peters Softweyr LLC [EMAIL PROTECTED] http://softweyr.com/ To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
Wes Peters wrote: Julian Elischer wrote: Arun Sharma wrote: On 26 Feb 2001 18:56:18 +0100, Matt Dillon [EMAIL PROTECTED] wrote: Ha. Right. Go through any piece of significant code and just see how much goes flying out the window because the code wants to simply assume things work. Then try coding conditionals all the way through to fix it... and don't forget you need to propogate the error condition back up the procedure chain too so the original caller knows why it failed. So, it all comes down to reimplementing the UNIX kernel in a language that supports exceptions, just like Linus suggested :) I've often considered writing a language SPECIFICALLY for writing the kernel. (no other uses) I mean it basically uses the same mechaninsims over and over and over again... linked lists, hash tables, nested loops, etc.etc. I'd like a language that lets me define the module I'm writing, define the way it should behave, and let the boring code be taken care of by itelf :-) Oh, like C++ STL? /me ducks and runs, trying not to gag. no, something specifically designed around kernel type of actions. declarations of "physical pointer", "kvm pointer" "User Pointer" for example, and being able to declare a structure (not 'struct') and say "this list is 'per process'" and have the list head automatically in the proc struct without haviong to add it there.. i.e backwards from today.. -- "Where am I, and what am I doing in this handbasket?" Wes Peters Softweyr LLC [EMAIL PROTECTED] http://softweyr.com/ -- __--_|\ Julian Elischer / \ [EMAIL PROTECTED] ( OZ) World tour 2000-2001 --- X_.---._/ v To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
In message [EMAIL PROTECTED], Julian Elischer writes: I still think that in such a case it should be possible to 'test the commitment' by touching all the allocated memory while trapping page faults. And what do you do if you get one? There's no undo button for SIGSEGV. Traditionally, you return from the signal handler right where you were. Can you get out of this with longjmp()? Probably. It's not exactly supported or guaranteed. In any event: 1. The C language spec doesn't require you to do this. 2. Other implementations have provided this guarantee, at least as an option. It's odd that I see lots of people arguing for segfaults killing the process accessing memory that has been "successfully" allocated, but no one arguing for the process getting killed when it exceeds a disk quota. -s To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
In message [EMAIL PROTECTED], Matt Dillon writes: The problem is a whole lot more complex then you think. Dealing with overcommit is not simply counting mapped pages, there are all sorts of issues involved. But the biggest gotcha is that putting in overcommit protection will not actually save your system from dying a terrible death. No, but it may allow me to stop and save some work and then gracefully suicide, rather than being segfaulted. I have seen plenty of designs which are able to say "give me another 32MB of memory. Oh, you can't? Okay, I'll do this the other way." Most importantly, though, I would rather be able to save my work and *THEN* abort because there's insufficient memory. Let's compare FreeBSD's handling of this situation to the handling we get under one of the most unstable pieces of crap imaginable: MacOS. When I run out of memory on my Mac, you know what happens? I get little warning boxes saying there isn't enough memory to complete a given operation. Almost all the time, I am then allowed to continue *using* the applications I have open. Indeed, the application which *gave* me the error message is still able to run... long enough for me to save my work and close a few windows. Apparently, the "desirable" behavior under FreeBSD would be for the application to believe it had memory, then suddenly abort without warning, and with no chance to save my work. It in fact makes it *more* likely that the system will die a horrible death, because with overcommit protection you wind up pre-reserving resources that would otherwise be reserved on demand (and often might not ever be used by the programs mapping those resources). Sure. But I want to know *UP FRONT* when I have problems. If I am trying to allocate too much memory, for God's sake, *TELL ME*. Give me warnings and errors. Make my program thrash, so I start wondering where the swap space went, and I can try to fix the problem, but don't just randomly kill processes that may or may not have done anything "wrong". Not only that, but overcommit protection does not protect you from thrashing, and quite often the system will become unusable long before it actually runs out of memory+swap. I am okay with saving my work slowly. I am not okay with not saving my work. A simple example of this is mmap(... PROT_READ|PROT_WRITE, MAP_PRIVATE), I am not sure that any promises have ever been made about the stability of mmap()'d memory. :) So before you get into tring to 'protect' yourself by implementing overcommit protection, you need to think long and hard on what you are actually protecting yourself from... and what you aren't. People seem to believe that edge cases such as allocating memory and then the system faulting the process later because it couldn't allocate the actual backing store later is of paramount importance but they seem to forget that by the time you reach that situation, you are generally already past the point of no return. This is quite possibly the case. On the other hand, in theory, the system could limit the total number of dirtyable pages such that it has a guarantee of enough space to swap... or at least promise to only kill programs that haven't had the "don't kill me" flag set. There are real applications, if only a few, where this would make a substantial difference. There are lots where it would at least provide warm fuzzies, and a checkbox conformance item that some one may have specified. Overcommit protection doesn't even come close to being able to prevent your system from getting into an unrecoverable situation and it certainly is no substitute for writing the software correctly (such that the memory failure can't occur in the first place). Unfortunately, as has been made abundantly clear, you can't write software such that a memory failure can't occur in the first place. If someone else writes a program that allocates a giant chunk of memory, then slowly goes through dirtying pages, it is quite likely that a correctly written program will eventually be killed. If, indeed, FreeBSD might also kill a program for dirtying bss space, you can't write *ANY* program "correctly"; you can always get a SIGSEGV for accessing an object that was in no way distinguishable from an object you have permission to access. -s To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
On Sat, 24 Feb 2001, Peter Seebach wrote: In message 9820.983050024@critter, Poul-Henning Kamp writes: I think there is a language thing you don't understand here. No, I just disagree. It is useful for the OS to provide a hook for memory which is *known to work* - and that is the environment C specifies. Send patches. Rik -- Linux MM bugzilla: http://linux-mm.org/bugzilla.shtml Virtual memory is like a game you can't win; However, without VM there's truly nothing to lose... http://www.surriel.com/ http://www.conectiva.com/ http://distro.conectiva.com/ To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
In message [EMAIL PROTECTED], Rik van Riel writes: No, I just disagree. It is useful for the OS to provide a hook for memory which is *known to work* - and that is the environment C specifies. Send patches. I may some day. It's not very high on my priority list; I'd probably try to fix UVM first. :) -s To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
[ Memory overcommit ] One important way to gain confidence that you're little box won't silently crash at the worst possible time for the customer is to be able to *prove* to yourself that it can't happen, given certain assumptions. Those assumptions usually include things like "the hardware is working properly" (e.g., no ECC errors) and "the compiler compiled my C code correctly". Given these basic assumptions, you go through and check that you've properly handled every possible case of input (malicious or otherwise) from the outside world. Part of the "proof" is verifying that you've checked all of your malloc(3) return values for NULL.. and assuming that if malloc(3) returns != NULL, then the memory is really there. Now, if malloc can return NULL and the memory *not* really be there, ^^^ I assume you meant 'can't' here, right? there is simply no way to prove that your code is not going to crash. Even in this case, there's no way to prove your code is not going to crash. The kernel has bugs, your software will have bugs (unless you've proved that it doesn't, and doing so on any significant piece of software will probably take longer to do than the amount of time you've spent writing and debugging it). And, what's to say that your correctly working software won't go bad right in the middle of your program running. There is no such thing as 100% fool-proof. This memory overcommit thing is the only case that I can think of where this happens, given the basic assumptions of correctly functioning hardware, etc. That is why it's especially annoying to (some) people. If you need 99.999% fool-proof, memory-overcommit can be one of the many classes of problems that bite you. However, in embededed systems, most folks design the system with particular software in mind. Therefore, you know ahead of time how much memory should be used, and can plan for how much memory is needed (overcommit or not) in your hardware design. (We're doing this right now in our 3rd generation product at work.) If the amount of memory is unknown (because of changing load conditions, and/or lack-of-experience with newer hardware), then overcommit *can* allow you to actually run 'better' than a non-overcommit system, though it doesn't necessarily give you the same kind of predictability when you 'hit the wall' like a non-overcommit system will do. Our embedded OS doesn't do memory-overcommit, but sometimes I wish it did, because it would give us some things for free. However, *IF* it did, we'd need some sort of mechanism (ie; AIX's SIGDANGER) that memory was getting tight, so the application could start dumping unused memory, or at least have an idea that something bad was happening so it could attempt to cleanup before it got whacked. :) Nate To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
In message [EMAIL PROTECTED], Nate Williams writes : Even in this case, there's no way to prove your code is not going to crash. Sure. But you can at least prove that all crashes are the result of bugs, not merely design "features". From the point of view of proving that a system is designed correctly, memory overcommit is a "bug". ;) -s To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
Even in this case, there's no way to prove your code is not going to crash. Sure. But you can at least prove that all crashes are the result of bugs, not merely design "features". 'Proving' something is correct is left as an excercise for the folks who have way too much time on their hand. At my previous job (SRI), we have folks who work full-time trying to prove algorithms. In general, proving out simple algorithms takes months, when the algorithm itself took 1-2 hours to design and write. Another thing is that crashes may have occurred because of invalid input, invalid output, valid but not expected input, etc... Again, memory overcommit is only *one* class of bugs that is avoided. The phrase "can't see the forest for the trees" jumps to mind. :) Nate To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
In message [EMAIL PROTECTED], Nate Williams writes : Again, memory overcommit is only *one* class of bugs that is avoided. The phrase "can't see the forest for the trees" jumps to mind. :) Sure, and likewise, bugs in libc are only one *class* of bugs we can avoid, but that doesn't mean we don't fix them. -s To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
:Matt Dillon wrote: : :.. : the system runs out of memory, even *with* overcommit protection. : In fact, all sorts of side effects occur even when the system :... : :That's an assumption. Ha. Right. Go through any piece of significant code and just see how much goes flying out the window because the code wants to simply assume things work. Then try coding conditionals all the way through to fix it... and don't forget you need to propogate the error condition back up the procedure chain too so the original caller knows why it failed. : : There ain't no magic bullet here, folks. By the time you get a memory : failure, even with no overcommit, it is far too late to save the day. : :Scientific computation. If, at one point, no more memory can be :allocated, you back off, save the present results, and try again later. This is irrelevant to the conversation. It has nothing to do with overcommit in the context it is being discussed. : :You assume too much. Quite a few of the more efficient garbage :collection algorithms depend on knowing when the memory has become full. This has nothing to do with overcommit in the context it is being discussed. In fact, this has nothing to do with OS memory management at all -- all garbage collected languages have their own infrastructure to determine when memory pressure requires collecting. :You keep allocating, and when malloc() returns NULL, *then* you run the :garbage collector, free some space, and try again. If malloc() doesn't :work, quite a few very efficient garbage collection algorithms become :impossible to implement. First, that's bullshit. Most garbage collection implementations require the memory 'size' to be hardwired. Second, that's bullshit because any program relying on that sort of operation had damn well better have its datasize limit set to something reasonable, or the garbage collector is going to run the system out of swap before it decides it needs to do a run through. :Just because *you* don't see how one thing can work, it doesn't mean it :can't work. As I have trivially shown above. You haven't shown anything above. Garbage collectors do not work that way and you really should know it. :Honestly, I think non-overcommit is a mistake and your approach is much :better, but it's not the only approach and there _are_ valid approaches :that depend on not overcommitting, and I really hate having to defend :non-overcommit against such bogus arguments. You've completely ignored the point that overcommit has nothing whatsoever to do with memory pressure. You are assuming that overcommit is some sort of magic bullet that will solve the memory pressure handling problem, and it is nothing of the sort. -Matt :Daniel C. Sobral (8-DCS) :[EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
I still think that in such a case it should be possible to 'test the commitment' by touching all the allocated memory while trapping page faults. and fault all your memory from 'potential' to 'allocated'. As someone said. it is not sure which program when you run out of swap, but I think you might be able to somehow change this behaviour to "the program making the request" fails (or gets a fault). You could allocate your memory. trap faults. touch all of the allocated memory. if it faults, you can remap some file to that location to allow the instruction to continue.. continue and abort the check.. exit as needed, OR continue with secure knowledge that all your memory is there. Alternatively you could allocate your own on-disk swapspace for a program by telling malloc to use a file for all your memory needs. I think the right way to implement this would be define a POSIX P1003.1 "mlockall(MCL_CURRENT|MCL_FUTURE)" along with a physical memory limit resource. I think this could be defined to give the requested malloc performance. This is defined to be not inherited so you'd still need to modify your program. I absolutely agree this is a can of worms, but this would be a way to proceed. Peter -- Peter Dufault ([EMAIL PROTECTED]) Realtime development, Machine control, HD Associates, Inc. Fail-Safe systems, Agency approval To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
In message [EMAIL PROTECTED], Matt Dillon writes: This has nothing to do with overcommit in the context it is being discussed. In fact, this has nothing to do with OS memory management at all -- all garbage collected languages have their own infrastructure to determine when memory pressure requires collecting. I think we were talking about C, and there are reasonable GC implementations for C... that assume that they can detect an out-of-memory condition because malloc returns a null pointer. First, that's bullshit. Most garbage collection implementations require the memory 'size' to be hardwired. Second, that's bullshit because any program relying on that sort of operation had damn well better have its datasize limit set to something reasonable, or the garbage collector is going to run the system out of swap before it decides it needs to do a run through. Fair enough, other circumstances might *also* cause a scan... but the fact remains that a GC system will want to know when it's out of memory, and will want some kind of warning other than a segfault. You've completely ignored the point that overcommit has nothing whatsoever to do with memory pressure. You are assuming that overcommit is some sort of magic bullet that will solve the memory pressure handling problem, and it is nothing of the sort. No one has said it solves all the problems. It solves *one specific* problem; the problem that you don't get *ANY* warning, or *ANY* chance to do anything, if you actually run out of available memory. Even if it's a transient failure that will go away in five minutes. Even if all you need to do is an fwrite and an fclose. -s To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
Peter Seebach wrote: It's odd that I see lots of people arguing for segfaults killing the process accessing memory that has been "successfully" allocated, but no one arguing for the process getting killed when it exceeds a disk quota. Disk quote is an artificial limit. If you remind each and every other time this discussion came up, you *can* set artificial memory limits, and that won't cause applications to be killed. But, of course, this particular solution you do not accept. Anyway, these are two very different situations, and comparing them is silly. If you want non-overcommit, code it and send the patches. -- Daniel C. Sobral(8-DCS) [EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED] Acabou o hipismo-arte. Mas a desculpa brasileira mais ouvida em Sydney e' que nao tem mais cavalo bobo por ai'. To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
Matt Dillon wrote: :Matt Dillon wrote: : :.. : the system runs out of memory, even *with* overcommit protection. : In fact, all sorts of side effects occur even when the system :... : :That's an assumption. Ha. Right. Go through any piece of significant code and just see how much goes flying out the window because the code wants to simply assume things work. Then try coding conditionals all the way through to fix it... and don't forget you need to propogate the error condition back up the procedure chain too so the original caller knows why it failed. sarcasmPerhaps you should re-acquaint yourself with exception handlers, as you seem to have forgotten them since you last worked with your C compiler. You know, the kind of thing where you can put a longjmp() in a proxy malloc(), and keep a setjmp() at the appropriate functional level so that if _any_ allocation fails down that path the whole function is cancelled?/sarcasm : : There ain't no magic bullet here, folks. By the time you get a memory : failure, even with no overcommit, it is far too late to save the day. : :Scientific computation. If, at one point, no more memory can be :allocated, you back off, save the present results, and try again later. This is irrelevant to the conversation. It has nothing to do with overcommit in the context it is being discussed. It has everything to do with overcommit in the context it is being discussed. I might be mistaken on exactly who, but I think it was Peter Seebach himself who once complained about his scientific application dieing suddenly under out of memory conditions on FreeBSD, though working perfectly on Solaris. Said application allocated memory if possible, and, if not, saved temporary results and went do something else until more memory became available. Now, you say "By the time you get a memory failure, even with no overcommit, it is far too late to save the day". This example shows that, no, you are wrong, it is not far too late. It is possible, at this point, to deal with the lack of more memory. I'll give you one more example. Protocol validation. It is often impossible to test all possible permutations of a protocol's dialog, but being able to go as deep as possible on execution tree and then, when you go out of memory, giving up on that path, backing down and continuing elsewhere let you get a partial validation, which is not enough to prove a protocol is correct but might well be enough to prove it is incorrect. This is a real application, and one in which an out of memory condition is not only handled but even expected. :You assume too much. Quite a few of the more efficient garbage :collection algorithms depend on knowing when the memory has become full. This has nothing to do with overcommit in the context it is being discussed. In fact, this has nothing to do with OS memory management at all -- all garbage collected languages have their own infrastructure to determine when memory pressure requires collecting. And, of course, those whose infrastructure depends on a malloc() returning NULL indicating the heap is full will not work on FreeBSD. (sarcasmYou do recall that many of these languages are written in C, don't you?/sarcasm) It has everything to do with overcommit. In this particular case, not only there _is_ something to do when the out of memory condition arise, but the very algorithm depends on it arising. And this is for some algorithms. Other algorithms require unbounded space for the collection but can handle out of memory conditions if they arise. :You keep allocating, and when malloc() returns NULL, *then* you run the :garbage collector, free some space, and try again. If malloc() doesn't :work, quite a few very efficient garbage collection algorithms become :impossible to implement. First, that's bullshit. Most garbage collection implementations require the memory 'size' to be hardwired. Second, that's bullshit Garbage Collection: Algorithms for Automatic Memory Management, Richard Jones and Rafael Lins. Bullshit is what you just said. because any program relying on that sort of operation had damn well better have its datasize limit set to something reasonable, or the garbage collector is going to run the system out of swap before it decides it needs to do a run through. Yes, indeed that happens, and dealing with swapped out objects is one topic of study in gc algorithms. For some applications, it would result in trashing. For others, most of the swapped out pages consist entirely of garbage. :Just because *you* don't see how one thing can work, it doesn't mean it :can't work. As I have trivially shown above. You haven't shown anything above. Garbage collectors do not work that way and you really should know it. You are obviously acquainted with an all too small subset of garbage collection algorithms. :Honestly, I
Re: Setting memory allocators for library functions.
: :... : : : :That's an assumption. : : Ha. Right. Go through any piece of significant code and just see how : much goes flying out the window because the code wants to simply assume : things work. Then try coding conditionals all the way through to fix : it... and don't forget you need to propogate the error condition back : up the procedure chain too so the original caller knows why it failed. : :sarcasmPerhaps you should re-acquaint yourself with exception :handlers, as you seem to have forgotten them since you last worked with :your C compiler. You know, the kind of thing where you can put a :longjmp() in a proxy malloc(), and keep a setjmp() at the appropriate :functional level so that if _any_ allocation fails down that path the :whole function is cancelled?/sarcasm sarcasm ... just try *proving* that sort of code. Good luck! I'm trying to imagine how to QA code that tries to deal with memory failures at any point and uses longjump, and I'm failing. :It has everything to do with overcommit in the context it is being :discussed. I might be mistaken on exactly who, but I think it was Peter :Seebach himself who once complained about his scientific application :dieing suddenly under out of memory conditions on FreeBSD, though :working perfectly on Solaris. Said application allocated memory if :possible, and, if not, saved temporary results and went do something :else until more memory became available. Said application was poorly written, then. Even on solaris if you actually run the system out of memory you can blow up other unrelated processes. To depend on that sort of operation is just plain dumb. At *best*, even on solaris, you can set the datasize limit for the process and try to manage memory that way. It is still a bad idea though because there's a chance that any libc call you make inside your core code, even something is innocuous as a printf(), may fail. The proper way to manage memory with this sort of application is to specify a hard limit in the program itself, and have the program keep track of its own useage and save itself off if it hits the hard limit. Alternatively you can monitor system load and install a signal handler to cause the program to save itself off and exit if the load gets too high... something that will occur long before the system actually runs out of memory. :I'll give you one more example. Protocol validation. It is often :impossible to test all possible permutations of a protocol's dialog, but :being able to go as deep as possible on execution tree and then, when :you go out of memory, giving up on that path, backing down and :continuing elsewhere let you get a partial validation, which is not :enough to prove a protocol is correct but might well be enough to prove :it is incorrect. This is a real application, and one in which an out of :memory condition is not only handled but even expected. This has nothing to do with memory overcommit. Nothing at all. What is your definition of out-of-memory? When swap runs out, or when the system starts to thrash? What is the point of running a scientific calculation if the machine turns into a sludge pile and would otherwise cause the calculation to take years to complete instead of days? You've got a whole lot more issues to deal with then simple memory overcommit, and you are ignoring them completely. :And, of course, those whose infrastructure depends on a malloc() :returning NULL indicating the heap is full will not work on FreeBSD. :(sarcasmYou do recall that many of these languages are written in C, :don't you?/sarcasm) Bullshit. If you care, a simple wrapper will do what you want. Modern systems tend to have huge amounts of swap. Depending on malloc to fail with unbounded resources in an overcommit OR a non-overcommit case is stupid, because the system will be thrashing heavily long before it even gets to that point. Depending on malloc() to fail by setting an appropriate datasize limit resource is more reasonable, and malloc() does work as expected if you do that. :It has everything to do with overcommit. In this particular case, not :only there _is_ something to do when the out of memory condition arise, :but the very algorithm depends on it arising. It has nothing to do with overcommit. You are confusing overcommit with hard-datasize limits, which can be set with a simple 'limit' command. : :Garbage Collection: Algorithms for Automatic Memory Management, Richard :Jones and Rafael Lins. Bullshit is what you just said. None of which requires overcommit. None of which would actually work in a real-world situation with or without overcommit if you do not hard-limit the memory resource for the program in the first place. You are again making the mistake of assuming that not having overcommit will magically solve
Re: Setting memory allocators for library functions.
In message [EMAIL PROTECTED], "Daniel C. Sobral" writes: Anyway, these are two very different situations, and comparing them is silly. They are situations in which an application can be killed and has no way to detect that it is about to do something wrong, and in which there *was* a correct way to notify the application of impending doom. Once we accept the argument "the C standard doesn't say this doesn't cause a segmentation fault", we can apply it to everything. -s To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
On Sun, 25 Feb 2001, Matt Dillon wrote: The problem is a whole lot more complex then you think. Dealing with overcommit is not simply counting mapped pages, there are all sorts of issues involved. But the biggest gotcha is that putting in overcommit protection will not actually save your system from dying a terrible death. It in fact makes it *more* likely that the system will die a horrible death, Indeed, but since a lot of the non-overcommit fans will not believe this, why not let them find out by themselves when they write a patch for it? And maybe, just maybe, they'll succeed in getting their idea of non-overcommit working with a patch which doesn't change dozens of places in the kernel and doesn't add any measurable overhead. (a while ago we had our yearly discussion on linux-kernel about this, after some time somebody showed up with code to do overcommit ... and, of course, the conclusion that it wouldn't work since he got to understand the problem better while writing the code ;)) regards, Rik -- Linux MM bugzilla: http://linux-mm.org/bugzilla.shtml Virtual memory is like a game you can't win; However, without VM there's truly nothing to lose... http://www.surriel.com/ http://www.conectiva.com/ http://distro.conectiva.com/ To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
On Tue, 27 Feb 2001, Daniel C. Sobral wrote: Matt Dillon wrote: :Matt Dillon wrote: : :.. : the system runs out of memory, even *with* overcommit protection. : In fact, all sorts of side effects occur even when the system :... : :That's an assumption. Ha. Right. Go through any piece of significant code and just see how much goes flying out the window because the code wants to simply assume things work. Then try coding conditionals all the way through to fix it... and don't forget you need to propogate the error condition back up the procedure chain too so the original caller knows why it failed. sarcasmPerhaps you should re-acquaint yourself with exception handlers, And just where are you going to grow the cache when the exception handler runs off the edge of the current stack page ? Rik -- Linux MM bugzilla: http://linux-mm.org/bugzilla.shtml Virtual memory is like a game you can't win; However, without VM there's truly nothing to lose... http://www.surriel.com/ http://www.conectiva.com/ http://distro.conectiva.com/ To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
In message [EMAIL PROTECTED], And maybe, just maybe, they'll succeed in getting their idea of non-overcommit working with a patch which doesn't change dozens of places in the kernel and doesn't add any measurable overhead. If it adds overhead, fine, make it a kernel option. :) Anyway, no, I'm not going to contribute code right now. If I get time to do this at all, I'll probably do it to UVM first. My main objection was to the claim that the C standard allows random segfaults. It doesn't. And yes, bad hardware is a conformance violation. :) -s To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
[hackers] Re: Setting memory allocators for library functions.
"Daniel" == Daniel C Sobral [EMAIL PROTECTED] writes: Daniel Dag-Erling Smorgrav wrote: None of these solutions are portable, however; Well, no, but the sole available definition of "portable" says that it is "portable" to assume that all the memory malloc can return is really available. Show me a modern OS (excluding real-time and/or embedded OSes) that makes this guarantee. Daniel Solaris and AIX (on AIX this is optional on a global or Daniel per-application level). IIRC, Digital-UNIX or OSF-1 ... or whatever it's called now. I seem to remember the first Alphas that arrived to a company I worked for had this set globally in the OS by default. Due to the bloat of the OS and Motif and other such things, they required simply amazing amounts of swap just to run. Dave. -- |David Gilbert, Velocet Communications. | Two things can only be | |Mail: [EMAIL PROTECTED] | equal if and only if they | |http://www.velocet.net/~dgilbert | are precisely opposite. | =GLO To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
On Mon, 26 Feb 2001, Peter Seebach wrote: In message [EMAIL PROTECTED], And maybe, just maybe, they'll succeed in getting their idea of non-overcommit working with a patch which doesn't change dozens of places in the kernel and doesn't add any measurable overhead. If it adds overhead, fine, make it a kernel option. :) Anyway, no, I'm not going to contribute code right now. If I get time to do this at all, I'll probably do it to UVM first. My main objection was to the claim that the C standard allows random segfaults. It doesn't. And yes, bad hardware is a conformance violation. :) I don't think a failed kernel-level allocation after overcommit should generate a segfault. IMHO it should send a bus error (or a sigkill if the process doesn't exit after the SIGBUS). Rationale: SIGSEGV for _user_ mistakes (process accesses wrong stuff) SIGBUS for _system_ errors (ECC error, kernel messes up, ...) cheers, Rik -- Linux MM bugzilla: http://linux-mm.org/bugzilla.shtml Virtual memory is like a game you can't win; However, without VM there's truly nothing to lose... http://www.surriel.com/ http://www.conectiva.com/ http://distro.conectiva.com/ To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
In message [EMAIL PROTECTED], Rik van Riel writes: I don't think a failed kernel-level allocation after overcommit should generate a segfault. IMHO it should send a bus error (or a sigkill if the process doesn't exit after the SIGBUS). Same difference, so far as the language is concerned. Rationale: SIGSEGV for _user_ mistakes (process accesses wrong stuff) SIGBUS for _system_ errors (ECC error, kernel messes up, ...) As long as we grant that it's the kernel *messing up*, I won't complain; no one said an implementation could be perfect, and known bugs go with the territory. I only object to attempts to portray it as a legitimate and correct implementation of the C spec. -s To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
In message [EMAIL PROTECTED],
Rik van Riel writes:
Rationale:
SIGSEGV for _user_ mistakes (process accesses wrong stuff)
SIGBUS for _system_ errors (ECC error, kernel messes up, ...)
Actually, this is not canonically the distinction made. On a Unix PC,
{
int *a, c[2];
char *b;
a = c;
b = a;
++b;
a = b;
*a = 0;
}
would get SIGBUS, because it was a bus error. The error is not a segmentation
fault; the memory written to is all legitimately available to the process. It
is a bus error, because the data access is not possible on the bus. :)
I think "the memory you thought you had actually doesn't exist anywhere" is
more like a segmentation fault than a bus error.
-s
To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-hackers" in the body of the message
Re: [hackers] Re: Setting memory allocators for library functions.
In message [EMAIL PROTECTED], Tony Finch writes: fork() with big data segments that cause swap to be reserved in case of a copy-on-write. The 2GB of swap is never actually used, but you still have to have it. That's a good point. So, we should warn people that asking for memory committments, having huge data spaces, and forking, is dangerous or stupid. :) -s To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
On 26 Feb 2001 18:56:18 +0100, Matt Dillon [EMAIL PROTECTED] wrote: Ha. Right. Go through any piece of significant code and just see how much goes flying out the window because the code wants to simply assume things work. Then try coding conditionals all the way through to fix it... and don't forget you need to propogate the error condition back up the procedure chain too so the original caller knows why it failed. So, it all comes down to reimplementing the UNIX kernel in a language that supports exceptions, just like Linus suggested :) -Arun To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
In message [EMAIL PROTECTED] Peter Seebach writes:
: I searched through the standard extensively to see if "allocates
: space" is defined and couldn't find anything other than 'the poitner
: can be used to access the space allocated.'
:
: EXACTLY!
:
: If it can't actually be used, then something has gone horribly wrong.
It says can, not must. I disagree with you that you can't overcommit.
Also, access is ill defined. Does that mean read access? write
access? How about execute access? It just says access and leaves the
meaning up to the implementor.
You fail to draw a distinction between "reading" the allocated area,
and "writing" to the allocated area. The standard makes no promises
that you can do both, and only has the vaguely defined access as to
what you can do with it.
Reasonable people can differ as to what this means.
: There appear to be no
: guarantees that you can always use all of the memory that you've
: allocated, the performance characterstics of accessing said memory or
: anything else.
:
: Performance characteristics are not at issue; I don't think anyone on the
: committee would complain about an implementation which provided "committed"
: space by backing it all to disk, all the time.
No one would. However, the standard does not preclude the ability to
overcommit. Many different systems do this already.
: Do not read too much into the above words. They mean exactly what
: they say. FreeBSD is in compliance.
:
: Assuming I make the Copenhagen meeting, I'll bring this up again as a DR
: or something. This gets debated every few years, and the consensus seems
: to be that malloc *MUST* return a valid pointer to space which can actually
: be used.
Really, I guess I missed those debates. I've not seen it in the
literature. I've not seen this addressed in the last 20 years that
systems have been overcomitting in any meaningful way.
: The space is indeed allocated to
: the process address space. System resource issues might get in the
: way of being able to use that, but that's true of anything you
: allocate.
:
: That's not what "allocate" means. A resource which is *allocated* is one
: you actually have access to.
Not necessarily. Allocate means, in the absense of resource
shortages, you get the memory. You might get the memory, you might
not. Consider a system that has a quota on dirty pages. If I malloc
a huge array, and then only use part of it, I can stay under my
quota. But if I use most of it by dirtying it, then the system can
kill the process.
: Basically, if your interpretation had been intended, the committee would have
: put in words to the effect of "access to an allocated object invokes undefined
: behavior".
:
: It doesn't.
Yes it does. You do not understand. If there's an ECC error, then
you do not have access to the memory without errors. This is no
different, just a different class of error.
: Therefore, the following code:
: #include stdlib.h
: int main(void) {
: unsigned char *p;
: if (2 SIZE_MAX) {
: p = malloc(2)
: if (p) {
: size_t i;
: for (i = 0; i 2; ++i) {
: p[i] = i % 5;
: }
: }
: }
: return 0;
: }
: *MUST* succeed and return 0. The program does not invoke undefined behavior;
: therefore, the compiler is obliged to ensure that it succeeds. It's fine for
: malloc to fail; it's not fine for the loop to segfault.
That's not necessarily true. There's nothing in the standard that
states that you have to be able to write to all parts of an allocated
region. It is a perfectly reasonable implementation to return the
memory from malloc. The system might enforce an upper bound on the
number of dirty pages a process has. malloc would not necessarily
have any way of knowing what those limits are, and those limits and
quotas might be dynamic or shared amoung classes of users. The limit
might be 100 right now, but 10 later. This is a host environment
issue. The malloc call gets a resource, but in the future you might
not be able to access that resource due to circumstances beyond the
control of the program.
Also, the program does not *HAVE* to succeed. If you have a low CPU
quota, for example, and the program exceeds it, the host environment
will signal SIGXCPU to the process. Another process might send this
one a SIGKILL. The system may crash in the middle of its execution.
There might be an unrecoverable memory parity error. There might be
an unrecoverable disk error on the swap partition. There might be any
number of things that preclude it from reaching completion, beyond
your control.
I don't see how this is significantly different than a dirty page
quota or a dirty page limit. That's what we're talking about here.
There's
Re: Setting memory allocators for library functions.
In message [EMAIL PROTECTED], Warner Losh writes: It says can, not must. But if it "can be used", and no one says "undefined behavior", then we're done; no undefined behavior is allowed. I disagree with you that you can't overcommit. Also, access is ill defined. It's defined better in C99 than it was in C89. Does that mean read access? write access? How about execute access? Within "defined behavior", read and write; all jumps to data are undefined behavior. The standard makes no promises that you can do both, and only has the vaguely defined access as to what you can do with it. Once again, if it had been intended to allow only limited access, it would say so. Reasonable people can differ as to what this means. Yes. : Performance characteristics are not at issue; I don't think anyone on the : committee would complain about an implementation which provided "committed" : space by backing it all to disk, all the time. No one would. However, the standard does not preclude the ability to overcommit. It is certainly read that way by a number of people, and I think we're currently a majority on the committee. No one cares; it's not as though an implementation is required to be in the maximally conforming mode at all times. Many different systems do this already. Yes, they do. That doesn't mean it's correct. I don't believe there's a single completely correct implementation of the fully ammended C89 standard, let alone C99. Really, I guess I missed those debates. I've not seen it in the literature. I've not seen this addressed in the last 20 years that systems have been overcomitting in any meaningful way. It's an old flamewar on comp.std.c. :) : That's not what "allocate" means. A resource which is *allocated* is one : you actually have access to. Not necessarily. Allocate means, in the absense of resource shortages, you get the memory. No. To allocate is to get. Allocation *fails* in the case of a resource shortage. You might get the memory, you might not. Consider a system that has a quota on dirty pages. If I malloc a huge array, and then only use part of it, I can stay under my quota. But if I use most of it by dirtying it, then the system can kill the process. Sure. Now imagine a quota on file I/O. If I start writing to a file, I'm under my quota, but if I exceed it, then the system can kill the process. If the standard had meant to say "access to allocated space invokes undefined behavior", it would have done so. Would you argue that the system which simply *kills* a process that tries to write to a full disk is correct? If not, can you show how that's any different? Yes it does. You do not understand. If there's an ECC error, then you do not have access to the memory without errors. This is no different, just a different class of error. Hardware errors do, indeed, produce a non-conforming implementation. However, on a system where the hardware works, overcommit-and-kill is *STILL* a conformance problem; killing on ECC errors isn't a conformance problem unless there are some. That's not necessarily true. There's nothing in the standard that states that you have to be able to write to all parts of an allocated region. Sure, and there's nothing in the standard that explicitly says "a call to frwrite with valid arguments should not raise SIGKILL", but everyone knows that would be a mistake. It is a perfectly reasonable implementation to return the memory from malloc. It's very reasonable, but it does not provide the environment described in the standard, which is an environment in which the memory returned by malloc is all available for use. might be 100 right now, but 10 later. This is a host environment issue. The malloc call gets a resource, but in the future you might not be able to access that resource due to circumstances beyond the control of the program. The same argument applies to killing programs when they write to a full disk. Also, the program does not *HAVE* to succeed. If you have a low CPU quota, for example, and the program exceeds it, the host environment will signal SIGXCPU to the process. Another process might send this one a SIGKILL. The system may crash in the middle of its execution. There might be an unrecoverable memory parity error. There might be an unrecoverable disk error on the swap partition. There might be any number of things that preclude it from reaching completion, beyond your control. Yes, and *EVERY ONE OF THOSE* is a conformance failure. I'm not saying we can produce an environment which is guaranteed to conform in all cases; it's obviously impossible. We *could* avoid breaking things. I don't see how this is significantly different than a dirty page quota or a dirty page limit. Very simple: We don't have control over hardware, and the Unix environment says you're allowed to do all sorts of things which, from the point of view of the spec, magically render the execution environment non-conforming.
Re: Setting memory allocators for library functions.
Dag-Erling Smorgrav wrote: [EMAIL PROTECTED] (Peter Seebach) writes: Is there any hope that, some day, a setting could be provided where a program could request that malloc *NOT* overcommit? There are programs which would rather know in advance, and clean up, than be killed abruptly. Malloc() does not overcommit - the kernel does. Malloc() doesn't know and doesn't care. I imagine you could add a compile-time option that forces obreak() (why do we still use brk()/sbrk()?) to prefault the newly allocated address space. Or you could write a malloc() implementation that only uses mmap(), and add a flag value that makes mmap() prefault pages, and fail if there isn't enough memory available. None of these solutions are portable, however; the portable way around memory overcommit is to write a malloc() wrapper that installs a SIGSEGV handler, then tries to dirty the newly allocated memory, and fails gracefully if this causes a segfault. Untested code: None of these solutions will work. In all of the above cases, it is still possible for an application to be killed due to out of memory conditions caused by other applications. The main point of non-overcommitting applications is to avoid that. Personally, I like AIX solution of a SIGDANGER signal, and having applications that do not have a handler installed be killed first. But that isn't what is being asked for. OTOH, the *only* way to get non-overcommit to FreeBSD is for someone who *wants* that feature to sit down and code it. It won't happen otherwise. -- Daniel C. Sobral(8-DCS) [EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED] Acabou o hipismo-arte. Mas a desculpa brasileira mais ouvida em Sydney e' que nao tem mais cavalo bobo por ai'. To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
Dag-Erling Smorgrav wrote: None of these solutions are portable, however; Well, no, but the sole available definition of "portable" says that it is "portable" to assume that all the memory malloc can return is really available. Show me a modern OS (excluding real-time and/or embedded OSes) that makes this guarantee. Solaris and AIX (on AIX this is optional on a global or per-application level). -- Daniel C. Sobral(8-DCS) [EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED] Acabou o hipismo-arte. Mas a desculpa brasileira mais ouvida em Sydney e' que nao tem mais cavalo bobo por ai'. To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
Julian Elischer wrote: to not be caught by surprise, simply touch every page after you allocate it. It doesn't work. The application killed by reason of insufficient resources is not (necessarily) the one that causes the page fault leading to that. Id est, if my application allocates all available memory, touching every page, and then some other application touches a pre-allocated page not backed by memory or swap, it is quite likely mine which will get killed. -- Daniel C. Sobral(8-DCS) [EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED] Acabou o hipismo-arte. Mas a desculpa brasileira mais ouvida em Sydney e' que nao tem mais cavalo bobo por ai'. To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
"Daniel C. Sobral" [EMAIL PROTECTED] writes: It doesn't work. The application killed by reason of insufficient resources is not (necessarily) the one that causes the page fault leading to that. This is arguably a bug which needs to be fixed. DES -- Dag-Erling Smorgrav - [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
In message [EMAIL PROTECTED], "Daniel C. Sobral" writes: OTOH, the *only* way to get non-overcommit to FreeBSD is for someone who *wants* that feature to sit down and code it. It won't happen otherwise. So, out of idle curiousity: If, somewhere down the road, I know the kernel well enough to attempt such a thing, what would the interest level be in merging such a feature? -s To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
In message [EMAIL PROTECTED] Peter Seebach writes: : In message [EMAIL PROTECTED], "Daniel C. Sobral" writes: : OTOH, the *only* way to get non-overcommit to FreeBSD is for someone who : *wants* that feature to sit down and code it. It won't happen otherwise. : : So, out of idle curiousity: If, somewhere down the road, I know the kernel : well enough to attempt such a thing, what would the interest level be in : merging such a feature? Assuming that it doesn't break anything, that it doesn't introduce a severe performance penalty and works, there would be interest. There are times that this is a desirable feature. Warner To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
Warner Losh writes: : OTOH, the *only* way to get non-overcommit to FreeBSD is for someone who : *wants* that feature to sit down and code it. It won't happen otherwise. : : So, out of idle curiousity: If, somewhere down the road, I know the kernel : well enough to attempt such a thing, what would the interest level be in : merging such a feature? Assuming that it doesn't break anything, that it doesn't introduce a severe performance penalty and works, there would be interest. There are times that this is a desirable feature. This thread reminds me of what happened when I brought up the same issue a few years ago, arguing that the kernel shouldn't overcommit memory (i.e., the same thing, everybody though I was nuts :) For me it helps to understand people's underlying motivation. Here's mine... my perspective probably comes from using FreeBSD in the embedded world, which is very different from using FreeBSD in the rack-mounted server world. One important way to gain confidence that you're little box won't silently crash at the worst possible time for the customer is to be able to *prove* to yourself that it can't happen, given certain assumptions. Those assumptions usually include things like "the hardware is working properly" (e.g., no ECC errors) and "the compiler compiled my C code correctly". Given these basic assumptions, you go through and check that you've properly handled every possible case of input (malicious or otherwise) from the outside world. Part of the "proof" is verifying that you've checked all of your malloc(3) return values for NULL.. and assuming that if malloc(3) returns != NULL, then the memory is really there. Now, if malloc can return NULL and the memory *not* really be there, there is simply no way to prove that your code is not going to crash. This memory overcommit thing is the only case that I can think of where this happens, given the basic assumptions of correctly functioning hardware, etc. That is why it's especially annoying to (some) people. -Archie __ Archie Cobbs * Packet Design * http://www.packetdesign.com To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
Archie Cobbs wrote: Warner Losh writes: : OTOH, the *only* way to get non-overcommit to FreeBSD is for someone who : *wants* that feature to sit down and code it. It won't happen otherwise. : : So, out of idle curiousity: If, somewhere down the road, I know the kernel : well enough to attempt such a thing, what would the interest level be in : merging such a feature? Assuming that it doesn't break anything, that it doesn't introduce a severe performance penalty and works, there would be interest. There are times that this is a desirable feature. This thread reminds me of what happened when I brought up the same issue a few years ago, arguing that the kernel shouldn't overcommit memory (i.e., the same thing, everybody though I was nuts :) For me it helps to understand people's underlying motivation. Here's mine... my perspective probably comes from using FreeBSD in the embedded world, which is very different from using FreeBSD in the rack-mounted server world. One important way to gain confidence that you're little box won't silently crash at the worst possible time for the customer is to be able to *prove* to yourself that it can't happen, given certain assumptions. Those assumptions usually include things like "the hardware is working properly" (e.g., no ECC errors) and "the compiler compiled my C code correctly". Given these basic assumptions, you go through and check that you've properly handled every possible case of input (malicious or otherwise) from the outside world. Part of the "proof" is verifying that you've checked all of your malloc(3) return values for NULL.. and assuming that if malloc(3) returns != NULL, then the memory is really there. Now, if malloc can return NULL and the memory *not* really be there, there is simply no way to prove that your code is not going to crash. This memory overcommit thing is the only case that I can think of where this happens, given the basic assumptions of correctly functioning hardware, etc. That is why it's especially annoying to (some) people. I still think that in such a case it should be possible to 'test the commitment' by touching all the allocated memory while trapping page faults. and fault all your memory from 'potential' to 'allocated'. As someone said. it is not sure which program when you run out of swap, but I think you might be able to somehow change this behaviour to "the program making the request" fails (or gets a fault). You could allocate your memory. trap faults. touch all of the allocated memory. if it faults, you can remap some file to that location to allow the instruction to continue.. continue and abort the check.. exit as needed, OR continue with secure knowledge that all your memory is there. Alternatively you could allocate your own on-disk swapspace for a program by telling malloc to use a file for all your memory needs. -Archie __ Archie Cobbs * Packet Design * http://www.packetdesign.com To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message -- __--_|\ Julian Elischer / \ [EMAIL PROTECTED] ( OZ) World tour 2000-2001 --- X_.---._/ v To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
:... : : merging such a feature? : : Assuming that it doesn't break anything, that it doesn't introduce a : severe performance penalty and works, there would be interest. There : are times that this is a desirable feature. : :This thread reminds me of what happened when I brought up the same :issue a few years ago, arguing that the kernel shouldn't overcommit :memory (i.e., the same thing, everybody though I was nuts :) : :For me it helps to understand people's underlying motivation. Here's The memory overcommit thread comes up once or twice a year. So this time around I am going to try to take a different tact in trying to explain the issue. One could argue about making the OS not overcommit until one is blue in the face. One could argue that every single routine that allocates memory must be prepared to handle a memory failure in a graceful manner. One could argue all sorts of high-and-mighty value things. But its all a crock. It simply isn't possible to gracefully handle an out of memory condition. All sorts of side effects occur when the system runs out of memory, even *with* overcommit protection. In fact, all sorts of side effects occur even when the system *doesn't* run out of memory, but instead just starts to thrash swap. All sorts of side effects occur if the system starts to get cornered memory-wise even if you don't *have* any swap. The number of possible combinations of effects is near infinite and nearly impossible to program against. Simply put, the 'overcommit' argument doesn't actually solve the problem in any meaningful way. Any significantly sized program that actually handled every possible combination and side effect of an out-of-memory condition gracefully would have so much conditional garbage cluttering it up that the rest of the code would be obscured in a haze of if()'s. There ain't no magic bullet here, folks. By the time you get a memory failure, even with no overcommit, it is far too late to save the day. There is only one correct way to handle an out of memory condition and that is to make sure it doesn't happen... so when it does happen you can treat it as a fatal error and scream bloody murder. It's a whole lot easier to design a program with bounded, deterministic memory use (e.g. database X requires Y kilobytes of memory per client instance) and control that use at the edges rather then to try to deal with memory failures gracefully in the deepest darkest corners of the program. And it's a whole lot more reliable, too. When I write such a program, if I care about bounding memory use I do it at the edges. I don't pollute low level allocation routines (like strdup(), small structural allocations, etc...) with all sorts of conditionals to gracefully back out of a memory failure. It's a huge waste of time. I just wrap the routines... safe_strdup(), safe_malloc(), safe_asprintf()... and the wrappers scream bloody murder and exit if they get a failure. It's that simple... and far more reliable. -Matt To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
:In message [EMAIL PROTECTED], "Daniel C. Sobral" writes: :OTOH, the *only* way to get non-overcommit to FreeBSD is for someone who :*wants* that feature to sit down and code it. It won't happen otherwise. : :So, out of idle curiousity: If, somewhere down the road, I know the kernel :well enough to attempt such a thing, what would the interest level be in :merging such a feature? : :-s The problem is a whole lot more complex then you think. Dealing with overcommit is not simply counting mapped pages, there are all sorts of issues involved. But the biggest gotcha is that putting in overcommit protection will not actually save your system from dying a terrible death. It in fact makes it *more* likely that the system will die a horrible death, because with overcommit protection you wind up pre-reserving resources that would otherwise be reserved on demand (and often might not ever be used by the programs mapping those resources). Not only that, but overcommit protection does not protect you from thrashing, and quite often the system will become unusable long before it actually runs out of memory+swap. This is true whether you have configured swap or not... it is perfectly easy to thrash a swapless system if the vast majority of pages are clean (e.g. you mmap() a lot of files read-only). A simple example of this is mmap(... PROT_READ|PROT_WRITE, MAP_PRIVATE), or even simply the static data and bss space associated with a program binary. Such memory only takes up physical space if you dirty it...you can *read* the memory all you want without having to reserve any resources, the OS can throw the physical page away at any time and reload it form the backing file. But the moment you dirty the memory the OS must allocate real resources to it ... either a physical page, or swap (and it can flip between the two). Many such mappings never actually require dirtying the underlying page and thus never actually take any resources. But with overcommit protection you have to reserve the resources immediately, even if the program will never use them. The result is that your system is likely to run out of memory long before it would without the overcommit protection. There are many other issues involved as well that are more complex, such as what you have to do when a program fork()s. So before you get into tring to 'protect' yourself by implementing overcommit protection, you need to think long and hard on what you are actually protecting yourself from... and what you aren't. People seem to believe that edge cases such as allocating memory and then the system faulting the process later because it couldn't allocate the actual backing store later is of paramount importance but they seem to forget that by the time you reach that situation, you are generally already past the point of no return. Overcommit protection doesn't even come close to being able to prevent your system from getting into an unrecoverable situation and it certainly is no substitute for writing the software correctly (such that the memory failure can't occur in the first place). -Matt To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
Matt Dillon wrote: But its all a crock. It simply isn't possible to gracefully handle an out of memory condition. All sorts of side effects occur when the system runs out of memory, even *with* overcommit protection. In fact, all sorts of side effects occur even when the system *doesn't* run out of memory, but instead just starts to thrash swap. All sorts of side effects occur if the system starts to get cornered memory-wise even if you don't *have* any swap. The number of possible combinations of effects is near infinite and nearly impossible to program against. Simply put, the 'overcommit' argument doesn't actually solve the problem in any meaningful way. Any significantly sized program that actually handled every possible combination and side effect of an out-of-memory condition gracefully would have so much conditional garbage cluttering it up that the rest of the code would be obscured in a haze of if()'s. That's an assumption. There ain't no magic bullet here, folks. By the time you get a memory failure, even with no overcommit, it is far too late to save the day. Scientific computation. If, at one point, no more memory can be allocated, you back off, save the present results, and try again later. There is only one correct way to handle an out of memory condition and that is to make sure it doesn't happen... so when it does happen you can treat it as a fatal error and scream bloody murder. It's a whole lot easier to design a program with bounded, deterministic memory use (e.g. database X requires Y kilobytes of memory per client instance) and control that use at the edges rather then to try to deal with memory failures gracefully in the deepest darkest corners of the program. And it's a whole lot more reliable, too. When I write such a program, if I care about bounding memory use I do it at the edges. I don't pollute low level allocation routines (like strdup(), small structural allocations, etc...) with all sorts of conditionals to gracefully back out of a memory failure. It's a huge waste of time. I just wrap the routines... safe_strdup(), safe_malloc(), safe_asprintf()... and the wrappers scream bloody murder and exit if they get a failure. It's that simple... and far more reliable. You assume too much. Quite a few of the more efficient garbage collection algorithms depend on knowing when the memory has become full. You keep allocating, and when malloc() returns NULL, *then* you run the garbage collector, free some space, and try again. If malloc() doesn't work, quite a few very efficient garbage collection algorithms become impossible to implement. Just because *you* don't see how one thing can work, it doesn't mean it can't work. As I have trivially shown above. Honestly, I think non-overcommit is a mistake and your approach is much better, but it's not the only approach and there _are_ valid approaches that depend on not overcommitting, and I really hate having to defend non-overcommit against such bogus arguments. You don't implement it because you don't think it's worth your time and you are not being paid to do so. That's a perfectly good reason, and if people would just stick with it the threads about this would die much sooner. -- Daniel C. Sobral(8-DCS) [EMAIL PROTECTED] [EMAIL PROTECTED] [EMAIL PROTECTED] Acabou o hipismo-arte. Mas a desculpa brasileira mais ouvida em Sydney e' que nao tem mais cavalo bobo por ai'. To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
[EMAIL PROTECTED] (Peter Seebach) writes: In message [EMAIL PROTECTED], Farooq Mela writes: How do you guys feel about this? It is a mistake to believe that you "don't have to worry about running out of memory". You should always check, every time, and exit as gracefully as you can. This is all academic since FreeBSD does memory overcommit, so unless you run out of address space for your process before you run out of actual memory and/or swap (not likely, but quite possible) malloc() will never return NULL and you won't know a thing until you dirty one page too many and segfault. DES -- Dag-Erling Smorgrav - [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
In message [EMAIL PROTECTED], Dag-Erling Smorgrav writes: This is all academic since FreeBSD does memory overcommit, so unless you run out of address space for your process before you run out of actual memory and/or swap (not likely, but quite possible) malloc() will never return NULL and you won't know a thing until you dirty one page too many and segfault. Is there any hope that, some day, a setting could be provided where a program could request that malloc *NOT* overcommit? There are programs which would rather know in advance, and clean up, than be killed abruptly. (To be pedantic, this is a conformance issue. If the memory isn't actually allocated, malloc shouldn't be returning a non-null pointer.) -s To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
[EMAIL PROTECTED] (Peter Seebach) writes:
Is there any hope that, some day, a setting could be provided where a program
could request that malloc *NOT* overcommit? There are programs which would
rather know in advance, and clean up, than be killed abruptly.
Malloc() does not overcommit - the kernel does. Malloc() doesn't know
and doesn't care.
I imagine you could add a compile-time option that forces obreak()
(why do we still use brk()/sbrk()?) to prefault the newly allocated
address space. Or you could write a malloc() implementation that only
uses mmap(), and add a flag value that makes mmap() prefault pages,
and fail if there isn't enough memory available.
None of these solutions are portable, however; the portable way around
memory overcommit is to write a malloc() wrapper that installs a
SIGSEGV handler, then tries to dirty the newly allocated memory, and
fails gracefully if this causes a segfault. Untested code:
#include errno.h
#include setjmp.h
#include signal.h
#include stdlib.h
#include string.h
static sigjmp_buf sigenv;
static void
sigsegv(int sig)
{
siglongjmp(sigenv, -1);
}
void *
mymalloc(size_t size)
{
sig_t saved_sig;
void *p;
if ((p = malloc(size)) == NULL)
return NULL;
if (sigsetjmp(env) == -1) {
free(p);
errno = ENOMEM;
return NULL;
}
saved_sig = signal(SIGSEGV, sigsegv);
bzero(p, size);
signal(SIGSEGV, saved_sig);
return p;
}
This probably won't work in threaded applications.
(To be pedantic, this is a conformance issue. If the memory isn't actually
allocated, malloc shouldn't be returning a non-null pointer.)
Address space is allocated, but initially unmapped. Actual pages
aren't faulted in until they're dirtied. Sometimes malloc() will give
you a chunk of memory that's already mapped and you'll be fine, but
sometimes (when allocating beyond what was previously used) it won't.
DES
--
Dag-Erling Smorgrav - [EMAIL PROTECTED]
To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
In message [EMAIL PROTECTED], Dag-Erling Smorgrav writes: Malloc() does not overcommit - the kernel does. Malloc() doesn't know and doesn't care. But it could still probably force the behavior. None of these solutions are portable, however; Well, no, but the sole available definition of "portable" says that it is "portable" to assume that all the memory malloc can return is really available. memory overcommit is to write a malloc() wrapper that installs a SIGSEGV handler, then tries to dirty the newly allocated memory, and fails gracefully if this causes a segfault. Ugh. Why not just have a flag for memory requests that requires the memory not to be overcommitted, and set it in "conforming mode"? The kernel *could* have memory which must not be overcommitted. -s To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
In message [EMAIL PROTECTED], Peter Seebach writes : In message [EMAIL PROTECTED], Dag-Erling Smorgrav writes: Malloc() does not overcommit - the kernel does. Malloc() doesn't know and doesn't care. But it could still probably force the behavior. None of these solutions are portable, however; Well, no, but the sole available definition of "portable" says that it is "portable" to assume that all the memory malloc can return is really available. No, this is not a guarantee. We also don't guarantee that you can read a file just because you managed to open it (disk errors, nfs servers going away, forced unmounts). -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 [EMAIL PROTECTED] | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
[EMAIL PROTECTED] (Peter Seebach) writes: In message [EMAIL PROTECTED], Dag-Erling Smorgrav writes: Malloc() does not overcommit - the kernel does. Malloc() doesn't know and doesn't care. But it could still probably force the behavior. Barring kernel changes, not easily, and only for single-threaded programs. None of these solutions are portable, however; Well, no, but the sole available definition of "portable" says that it is "portable" to assume that all the memory malloc can return is really available. Show me a modern OS (excluding real-time and/or embedded OSes) that makes this guarantee. memory overcommit is to write a malloc() wrapper that installs a SIGSEGV handler, then tries to dirty the newly allocated memory, and fails gracefully if this causes a segfault. Ugh. Why not just have a flag for memory requests that requires the memory not to be overcommitted, and set it in "conforming mode"? Read the paragraph that precedes the one you're quoting. DES -- Dag-Erling Smorgrav - [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
In message 9402.983047348@critter, Poul-Henning Kamp writes: Well, no, but the sole available definition of "portable" says that it is "portable" to assume that all the memory malloc can return is really available. No, this is not a guarantee. Yes, it is. If the memory isn't available, malloc returns NULL. We also don't guarantee that you can read a file just because you managed to open it (disk errors, nfs servers going away, forced unmounts). Sure. And all the IO functions have return codes. -s To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
In message [EMAIL PROTECTED], Peter Seebach writes : In message 9402.983047348@critter, Poul-Henning Kamp writes: Well, no, but the sole available definition of "portable" says that it is "portable" to assume that all the memory malloc can return is really available. No, this is not a guarantee. Yes, it is. If the memory isn't available, malloc returns NULL. The guarantee is "If malloc returns NULL there is no memory you can use". That doesn't mean that just because != NULL is returned that memory will in fact be available. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 [EMAIL PROTECTED] | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
In message 9469.983047707@critter, Poul-Henning Kamp writes: The guarantee is "If malloc returns NULL there is no memory you can use". No, it's "if the memory is not available, malloc returns NULL". That doesn't mean that just because != NULL is returned that memory will in fact be available. Sure, and access to stdin can segfault too. If we want a decent quality of implementation for C, malloc can't lie. -s To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
Poul-Henning Kamp wrote: In message [EMAIL PROTECTED], Peter Seebach writes : In message 9402.983047348@critter, Poul-Henning Kamp writes: Well, no, but the sole available definition of "portable" says that it is "portable" to assume that all the memory malloc can return is really available. No, this is not a guarantee. Yes, it is. If the memory isn't available, malloc returns NULL. The guarantee is "If malloc returns NULL there is no memory you can use". That doesn't mean that just because != NULL is returned that memory will in fact be available. If the intended behaviour of malloc is that it returns a pointer to memory that is allocated but which may not be available when accessed, the man page needs to be corrected to make this defect in the implementation clear. To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
In message [EMAIL PROTECTED], Peter Seebach writes : In message 9469.983047707@critter, Poul-Henning Kamp writes: The guarantee is "If malloc returns NULL there is no memory you can use". No, it's "if the memory is not available, malloc returns NULL". No it is not and it never was. See RFC748 for why you cannot possibly be right. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 [EMAIL PROTECTED] | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
In message 9631.983048712@critter, Poul-Henning Kamp writes: No it is not and it never was. The C committee believes it is. -s To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
In message [EMAIL PROTECTED], Peter Seebach writes : In message 9631.983048712@critter, Poul-Henning Kamp writes: No it is not and it never was. The C committee believes it is. The C committee doesn't deal with operating systems or the real world for that matter. They only interest them selves with a programming language. I think there is a language thing you don't understand here. If a car is advertised as "up to 18 km/l" that means that they will guarantee that you will never ever see it doing better. It doesn't mean that you will ever see that level of performance. Like wise malloc(3) will never return a pointer which it _knows_ you cannot use, but it may not have the information that enables it to determine that you cannot use it and thus unknowingly pass the pointer to you, even though you cannot use it. EOS from here. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 [EMAIL PROTECTED] | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
In message 9820.983050024@critter, Poul-Henning Kamp writes: I think there is a language thing you don't understand here. No, I just disagree. It is useful for the OS to provide a hook for memory which is *known to work* - and that is the environment C specifies. -s To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
In message [EMAIL PROTECTED], Peter Seebach writes : In message 9820.983050024@critter, Poul-Henning Kamp writes: I think there is a language thing you don't understand here. No, I just disagree. It is useful for the OS to provide a hook for memory which is *known to work* - and that is the environment C specifies. And just how will the OS know that a particular memory chip will not generate an uncorrectable ECC error ? Did you read the RFC I pointed you at ? -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 [EMAIL PROTECTED] | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
[EMAIL PROTECTED] (Peter Seebach) writes: In message 9877.983050267@critter, Poul-Henning Kamp writes: And just how will the OS know that a particular memory chip will not generate an uncorrectable ECC error ? It can't, but that's no reason for the OS to *add* reasons for failures. It doesn't (and I'm very close to using expletives here). On the contrary, it tries to always satisfy the application's requests and hopes for the best. It's quite possible that memory isn't available when you call malloc(), but becomes available before you actually get to dirty the pages that were allocated. DES -- Dag-Erling Smorgrav - [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
In message [EMAIL PROTECTED], Dag-Erling Smorgrav writes: It doesn't (and I'm very close to using expletives here). On the contrary, it tries to always satisfy the application's requests and hopes for the best. Yes, but it could provide a stronger guarantee than it does. It's quite possible that memory isn't available when you call malloc(), but becomes available before you actually get to dirty the pages that were allocated. Sure. And for many applications, that's a win. For some, though, the *CHANCE* of overcommitting killing the process is a serious problem, and the application would be better off if it could warn the user that insufficient memory is available, and perhaps more should be provided. It would be a useful option. I'm not saying it's the *only* useful option, or even always the best. However, it should be trivial enough to provide this option, and it *does* serve a useful purpose. -s To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
Peter Seebach wrote: In message [EMAIL PROTECTED], Dag-Erling Smorgrav writes: This is all academic since FreeBSD does memory overcommit, so unless you run out of address space for your process before you run out of actual memory and/or swap (not likely, but quite possible) malloc() will never return NULL and you won't know a thing until you dirty one page too many and segfault. Is there any hope that, some day, a setting could be provided where a program could request that malloc *NOT* overcommit? There are programs which would rather know in advance, and clean up, than be killed abruptly. to not be caught by surprise, simply touch every page after you allocate it. (To be pedantic, this is a conformance issue. If the memory isn't actually allocated, malloc shouldn't be returning a non-null pointer.) -s To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message -- __--_|\ Julian Elischer / \ [EMAIL PROTECTED] ( OZ) World tour 2000-2001 --- X_.---._/ v To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
In message [EMAIL PROTECTED] Peter Seebach writes: : In message 9631.983048712@critter, Poul-Henning Kamp writes: : No it is not and it never was. : : The C committee believes it is. I do not believe you. Such a "believe" does not appear to be embodied in the C-99 standard. The standard (well draft standard) says: 7.20.3.3 The malloc function Synopsis [#1] #include stdlib.h void *malloc(size_t size); Description [#2] The malloc function allocates space for an object whose size is specified by size and whose value is indeterminate. Returns [#3] The malloc function returns either a null pointer or a pointer to the allocated space. Notice that no guarantees about the ability to use all the space, or that it is somehow guaranteed to be there. It just says that that space is allocated. Nothing more and nothing less. I defy you to quote me someting from the standard that is contrary to my position. I searched through the standard extensively to see if "allocates space" is defined and couldn't find anything other than 'the poitner can be used to access the space allocated.' There appear to be no guarantees that you can always use all of the memory that you've allocated, the performance characterstics of accessing said memory or anything else. Do not read too much into the above words. They mean exactly what they say. FreeBSD is in compliance. The space is indeed allocated to the process address space. System resource issues might get in the way of being able to use that, but that's true of anything you allocate. Warner To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
In message [EMAIL PROTECTED], Warner Losh writes:
I do not believe you. Such a "believe" does not appear to be embodied
in the C-99 standard.
It's somewhat debated, but it has come up in discussions, and the consensus
is that, if malloc returns a pointer, the memory has to be usable.
Notice that no guarantees about the ability to use all the space, or
that it is somehow guaranteed to be there.
That's what space *is*. Space is something that can store values. If it
can't actually store values, it's not space.
space is allocated. Nothing more and nothing less. I defy you to
quote me someting from the standard that is contrary to my position.
Heh.
I searched through the standard extensively to see if "allocates
space" is defined and couldn't find anything other than 'the poitner
can be used to access the space allocated.'
EXACTLY!
If it can't actually be used, then something has gone horribly wrong.
There appear to be no
guarantees that you can always use all of the memory that you've
allocated, the performance characterstics of accessing said memory or
anything else.
Performance characteristics are not at issue; I don't think anyone on the
committee would complain about an implementation which provided "committed"
space by backing it all to disk, all the time.
Do not read too much into the above words. They mean exactly what
they say. FreeBSD is in compliance.
Assuming I make the Copenhagen meeting, I'll bring this up again as a DR
or something. This gets debated every few years, and the consensus seems
to be that malloc *MUST* return a valid pointer to space which can actually
be used.
The space is indeed allocated to
the process address space. System resource issues might get in the
way of being able to use that, but that's true of anything you
allocate.
That's not what "allocate" means. A resource which is *allocated* is one
you actually have access to.
Basically, if your interpretation had been intended, the committee would have
put in words to the effect of "access to an allocated object invokes undefined
behavior".
It doesn't.
Therefore, the following code:
#include stdlib.h
int main(void) {
unsigned char *p;
if (2 SIZE_MAX) {
p = malloc(2)
if (p) {
size_t i;
for (i = 0; i 2; ++i) {
p[i] = i % 5;
}
}
}
return 0;
}
*MUST* succeed and return 0. The program does not invoke undefined behavior;
therefore, the compiler is obliged to ensure that it succeeds. It's fine for
malloc to fail; it's not fine for the loop to segfault.
This is probably one of the single most-unimplemented features in the spec,
but it really wouldn't be that hard to provide as an option.
-s
To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
* Farooq Mela [EMAIL PROTECTED] [010222 23:52] wrote: This is not what I am arguing. I gave a simple example of an xmalloc function which does just print an error and exit. However, I've written many large applications using this sort of scheme where when allocation fails, all sorts of cleanup is performed before the process exits (cleaning up restoring terminal mode, gracefully closing files / sockets / etc, syslogging the event, etc). It is hardly ever a simple exit as soon as allocation fails. Here's exactly what's wrong with your idea: Some internal libc function that _can_ gracefully handle an out of memory return will not be able to do so if your malloc wrappers wrest control out from under them. Honestly, any code is somewhat flawed if it doesn't take extra care not to have sections where an abrupt shutdown can cause a lot of damage or inability to recover on restart. However... Some internal libc functions may attempt an allocation while in the midst of doing something evil to your program such as fiddling with signal masks or timers, if you get your "out of memory" callback and the libc function hasn't had time to restore your normal program context, you're going to have a world of pain to deal with. I can't provide any specific examples of this potentially happening, but I can imagine it being a problem, especially deep within something like the userland thread library or other complex library. If you need to deal with asprintf problems, then make an xasprinf that does the callback in your context, not from deep within libc. -Alfred To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
Alfred Perlstein wrote: * Farooq Mela [EMAIL PROTECTED] [010222 23:52] wrote: This is not what I am arguing. I gave a simple example of an xmalloc function which does just print an error and exit. However, I've written many large applications using this sort of scheme where when allocation fails, all sorts of cleanup is performed before the process exits (cleaning up restoring terminal mode, gracefully closing files / sockets / etc, syslogging the event, etc). It is hardly ever a simple exit as soon as allocation fails. Here's exactly what's wrong with your idea: Some internal libc function that _can_ gracefully handle an out of memory return will not be able to do so if your malloc wrappers wrest control out from under them. Honestly, any code is somewhat flawed if it doesn't take extra care not to have sections where an abrupt shutdown can cause a lot of damage or inability to recover on restart. However... Some internal libc functions may attempt an allocation while in the midst of doing something evil to your program such as fiddling with signal masks or timers, if you get your "out of memory" callback and the libc function hasn't had time to restore your normal program context, you're going to have a world of pain to deal with. I can't provide any specific examples of this potentially happening, but I can imagine it being a problem, especially deep within something like the userland thread library or other complex library. If you need to deal with asprintf problems, then make an xasprinf that does the callback in your context, not from deep within libc. I agree. Some instances of malloc failure can be / must be dealt with gracefully in libc. I am not saying that ALL usages of malloc would be replaced with calling the user-settable allocator. Only those that are not modifying, as you mentioned, the process' signal disposition and other such attributes, and *certainly* not from within the thread library. Anyway, doesnt seem like this idea is about to fly. I guess I'll shut up now. :-) -Farooq To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
* Farooq Mela [EMAIL PROTECTED] [010223 00:22] wrote:
Alfred Perlstein wrote:
* Farooq Mela [EMAIL PROTECTED] [010222 23:52] wrote:
This is not what I am arguing. I gave a simple example of an xmalloc
function which does just print an error and exit. However, I've written
many large applications using this sort of scheme where when allocation
fails, all sorts of cleanup is performed before the process exits
(cleaning up restoring terminal mode, gracefully closing files /
sockets / etc, syslogging the event, etc). It is hardly ever a simple
exit as soon as allocation fails.
Here's exactly what's wrong with your idea:
Some internal libc function that _can_ gracefully handle an out
of memory return will not be able to do so if your malloc wrappers
wrest control out from under them.
Honestly, any code is somewhat flawed if it doesn't take extra care
not to have sections where an abrupt shutdown can cause a lot of
damage or inability to recover on restart. However...
Some internal libc functions may attempt an allocation while in the
midst of doing something evil to your program such as fiddling with
signal masks or timers, if you get your "out of memory" callback
and the libc function hasn't had time to restore your normal program
context, you're going to have a world of pain to deal with.
I can't provide any specific examples of this potentially happening,
but I can imagine it being a problem, especially deep within something
like the userland thread library or other complex library.
If you need to deal with asprintf problems, then make an xasprinf
that does the callback in your context, not from deep within libc.
I agree. Some instances of malloc failure can be / must be dealt with
gracefully in libc. I am not saying that ALL usages of malloc would be
replaced with calling the user-settable allocator. Only those that are
not modifying, as you mentioned, the process' signal disposition and
other such attributes, and *certainly* not from within the thread
library.
Anyway, doesnt seem like this idea is about to fly. I guess I'll shut up
now. :-)
Well, while looking at actually doing what you're saying, phk malloc
will call a function (from the manpage):
void
(*_malloc_message)(char *p1, char *p2, char *p3, char *p4)
you can set _malloc_message = your callback.
the only problem is that the failure codes don't seem to be
enumerated, they seem to be strings, if there was a way to test:
/*
* this is untested
*/
void (*saved_malloc_callback)(char *p1, char *p2, char *p3, char *p4);
void
my_malloc_callback(char *p1, char *p2, char *p3, char *p4)
{
if (p1 == malloc_out_of_memory_str) {
/* do out of memory handling */
} else {
saved_malloc_callback(p1, p2, p3, p4);
}
}
int
main (void) {
saved_malloc_callback = _malloc_message;
_malloc_message = my_malloc_callback;
more_main();
}
or something, you could then hijack _malloc_message for your
callback.
Of course this would be highly FreeBSD specific, but so would the
modification you wanted in the first place.
Poul-Henning, is there any way to do a valid test to determine that
the '_malloc_message' callback parameter means "out of memory"?
I'm pretty sure that we should discourage anyone wanting to abuse
the interface by doing this, but it does seem sort of interesting.
:)
--
-Alfred Perlstein - [[EMAIL PROTECTED]|[EMAIL PROTECTED]]
To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
Farooq Mela [EMAIL PROTECTED] wrote:
Hi,
Usually when I write programs, I have functions such as the following:
void *
xmalloc(size_t size)
{
nice code
}
void *
xrealloc(void *ptr, size_t size)
{
nice code
}
And then I use these instead of malloc and realloc throughout the
program, and never have to worry about return values. Sometimes these
functions also have additional cleanup they perform. Anyway, there are
certain libc functions which also use malloc, such as getaddrinfo,
vasprintf, etc. These may fail with errno = ENOMEM. In general, it is
annoying to have to check the return values for these functions too.
Would it not be convenient to set the memory allocator used by certain
functions inside libc? I.E, be able to write:
set_allocator(xmalloc);
set_reallocator(xrealloc);
From then on, one would never have to worry about the functions running
out of memory. I know that wrappers for functions which allocate memory
can also be written, but I feel that my proposal would is in general a
more convenient solution.
How do you guys feel about this?
-Farooq
This would have probably been an outstanding idea when the
C standard was being put together... (and, who knows, somethine
similar may very well have been proposed.)
But, let me point out that adding such a feature to the FreeBSD
library doesn't mean you can dispense with your checks and/or
wrapping functions. As soon as your code needs to run on another
platform, you'll need those checks...
Such is the way of the world - when you have a standard, that's
all you can expect from other systems...
- Dave Rivers -
--
[EMAIL PROTECTED] Work: (919) 676-0847
Get your mainframe (370) `C' compiler at http://www.dignus.com
To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-hackers" in the body of the message
Setting memory allocators for library functions.
Hi,
Usually when I write programs, I have functions such as the following:
void *
xmalloc(size_t size)
{
void *p;
if ((p=malloc(size))==NULL) {
fprintf(stderr, "out of memory\n");
exit(1);
}
return(p);
}
void *
xrealloc(void *ptr, size_t size)
{
void *p;
if ((p=realloc(ptr,size))==NULL) {
fprintf(stderr, "out of memory\n");
exit(1);
}
return(p);
}
And then I use these instead of malloc and realloc throughout the
program, and never have to worry about return values. Sometimes these
functions also have additional cleanup they perform. Anyway, there are
certain libc functions which also use malloc, such as getaddrinfo,
vasprintf, etc. These may fail with errno = ENOMEM. In general, it is
annoying to have to check the return values for these functions too.
Would it not be convenient to set the memory allocator used by certain
functions inside libc? I.E, be able to write:
set_allocator(xmalloc);
set_reallocator(xrealloc);
From then on, one would never have to worry about the functions running
out of memory. I know that wrappers for functions which allocate memory
can also be written, but I feel that my proposal would is in general a
more convenient solution.
How do you guys feel about this?
-Farooq
To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
In message [EMAIL PROTECTED], Farooq Mela writes: How do you guys feel about this? It is a mistake to believe that you "don't have to worry about running out of memory". You should always check, every time, and exit as gracefully as you can. -s To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
Peter Seebach wrote:
It is a mistake to believe that you "don't have to worry about running
out of memory". You should always check, every time, and exit as gracefully
as you can.
-s
Of course I realize that allocating memory can fail. That is why I use
xmalloc and friends - so that I can avoid having to check for failure
each time I want to allocate memory. I don't believe you understand what
I am proposing. Consider the following inside libc:
void *(*libc_malloc)(size_t)=malloc;
void *(*libc_calloc)(size_t, size_t)=calloc;
void *(*libc_realloc)(void *, size_t)=realloc;
void
set_allocator(void *(*func)(size_t))
{
libc_malloc=func;
}
/* Same type of functions for calloc and realloc.. */
Now, consider some other function that is part of libc, such as
getaddrinfo. Say it does the following:
/* code ... */
foo=malloc(128);
if (foo==NULL) {
/* do clean-up.. */
errno=ENOMEM;
return(NULL);
}
This would be replaced by:
foo=(*libc_malloc)(128);
if (foo=NULL) {
/* do clean-up */
errno=ENOMEM;
return(NULL);
}
It will still have to check for the allocation failing. But if this were
to be done, an application which installs its own allocators will not
have to worry about anything inside libc running out of memory, and will
not have to handle that error condition. Furthermore, inside the
user-defined allocator (xmalloc etc), other types of cleanup can be
handled if the real malloc() returns NULL. (Atexit can only do so much.)
Hope this clears it up.
-Farooq
To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
In message [EMAIL PROTECTED], Farooq Mela writes: Of course I realize that allocating memory can fail. That is why I use xmalloc and friends - so that I can avoid having to check for failure each time I want to allocate memory. That's the problem. You still *NEED* to check. You can't just exit; you have to figure out what you have open, and close it. You have to figure out what files you may have been halfway through writing, and make sure that you aren't about to leave a critical file half-written, with the only copy of the rest of the data somewhere in memory. It is very, very, rarely correct to say "oops, no memory, I will quit immediately without saving any work". Imagine a word processor with this behavior. No attempt to save your file, no nothing, it just suddenly closes all windows and dies. There are very few programs where it is acceptable to abort just because you ran out of memory. I don't believe you understand what I am proposing. I do, lots of programs do it. It's a bad idea. You really still do have to check, because in almost all cases, there will be stuff you should do on your way down, beyond just "exit". It will still have to check for the allocation failing. But if this were to be done, an application which installs its own allocators will not have to worry about anything inside libc running out of memory, and will not have to handle that error condition. Of course it will! It can't guarantee that the memory is available, and it can't guarantee that it cleans up properly, because "proper cleanup" may vary from one place to another. Furthermore, inside the user-defined allocator (xmalloc etc), other types of cleanup can be handled if the real malloc() returns NULL. (Atexit can only do so much.) Exactly. Atexit *can't do enough*. Each individual place where you do something that *could* fail must be checked, and the failure handled correctly *for that place*. You can't short-cut this; if you do, your code will inevitably fail in the most inconvenient possible way, because computers are ornery. Yes, it's a lot of work checking for every single error that could occur, and handling it in a graceful fashion. However, it is *MUCH* better for, say, gethostbyname to set errno to ENOMEM and return a null pointer, than for gethostbyname to ritually suicide the entire program. The allocator which spews an error and dies is almost always an incorrect allocator. There may be counterexamples, but they're rare, and since all of the relevant library functions can probably fail *FOR OTHER REASONS*, you still have to check all the return values and handle them yourself. Even if you can prove that getaddrinfo can never fail because malloc failed, it can still fail for other reasons. You still have to check for it failing. "Fixing" the allocator doesn't remove the need to check the return of every function that can fail. -s To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
Re: Setting memory allocators for library functions.
Peter Seebach wrote: In message [EMAIL PROTECTED], Farooq Mela writes: Of course I realize that allocating memory can fail. That is why I use xmalloc and friends - so that I can avoid having to check for failure each time I want to allocate memory. That's the problem. You still *NEED* to check. You can't just exit; you have to figure out what you have open, and close it. You have to figure out what files you may have been halfway through writing, and make sure that you aren't about to leave a critical file half-written, with the only copy of the rest of the data somewhere in memory. It is very, very, rarely correct to say "oops, no memory, I will quit immediately without saving any work". This is not what I am arguing. I gave a simple example of an xmalloc function which does just print an error and exit. However, I've written many large applications using this sort of scheme where when allocation fails, all sorts of cleanup is performed before the process exits (cleaning up restoring terminal mode, gracefully closing files / sockets / etc, syslogging the event, etc). It is hardly ever a simple exit as soon as allocation fails. [snip] There are very few programs where it is acceptable to abort just because you ran out of memory. I don't believe you understand what I am proposing. I do, lots of programs do it. It's a bad idea. You really still do have to check, because in almost all cases, there will be stuff you should do on your way down, beyond just "exit". It will still have to check for the allocation failing. But if this were to be done, an application which installs its own allocators will not have to worry about anything inside libc running out of memory, and will not have to handle that error condition. Of course it will! It can't guarantee that the memory is available, and it can't guarantee that it cleans up properly, because "proper cleanup" may vary from one place to another. What I mean here is that the *application* will not have to handle that error condition. It will be handled by the the malloc wrapper(s). Furthermore, inside the user-defined allocator (xmalloc etc), other types of cleanup can be handled if the real malloc() returns NULL. (Atexit can only do so much.) Exactly. Atexit *can't do enough*. Each individual place where you do something that *could* fail must be checked, and the failure handled correctly *for that place*. You can't short-cut this; if you do, your code will inevitably fail in the most inconvenient possible way, because computers are ornery. Yes, it's a lot of work checking for every single error that could occur, and handling it in a graceful fashion. However, it is *MUCH* better for, say, gethostbyname to set errno to ENOMEM and return a null pointer, than for gethostbyname to ritually suicide the entire program. The allocator which spews an error and dies is almost always an incorrect allocator. There may be counterexamples, but they're rare, and since all of the relevant library functions can probably fail *FOR OTHER REASONS*, you still have to check all the return values and handle them yourself. Yes, but you would not have to write code which checks for ENOMEM. I will bet that the error handling code you write for getcwd giving ENOENT is very different from the code which handles ENOMEM from getcwd. I never write programs in which I must check the return value of each and every call to malloc/realloc/etc. It would be a lot of repetitive code. That is exactly why I use malloc wrappers, which then do all the required cleanup if allocation fails. It is actually a very effective system if done correctly, and makes life easier for me at least ;-). I would rather not have to check the return value from something like memory allocation, which is done very often in programs, and in todays VM systems, doesnt fail until all swap space is consumed. Even if you can prove that getaddrinfo can never fail because malloc failed, it can still fail for other reasons. You still have to check for it failing. "Fixing" the allocator doesn't remove the need to check the return of every function that can fail. Indeed. Getaddrinfo is an example of a function that can fail for other reasons. But something such as asprintf or getcwd, etc, it is annoying to have to write code which will perform the cleanup that a malloc wrapper would otherwise execute. I just believe it would be a convenience to programmers if this system could be implemented. If someone prefers not to use it, and check each return value, that is fine; they are not required to use it. -Farooq To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-hackers" in the body of the message
